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ABSTRACT 


This  report  presents  the  results  of  a  study  for  the  Naval  Sea  Systems 
Command  conducted  in  support  of  the  development  of  a  Platform  Adaptor  Group 
(PAG)  for  the  Joint  Tactical  Information  Distribution  System  (JTIDS) . 


SUMMARY 


The  Joint  Tactical  Information  Distribution  System  (JTIDS)  is  an  advanced 
information  distribution  system  that  provides  communications,  navigation, 
and  identif  ^tion  (CNI)  capabilities  in  an  integrated  form  for  application 
to  military  and  air  defense  operations.  This  study  for  the  Naval  Sea 
Systems  Command  (NAVSEA)  was  conducted  in  support  of  the  development  of  a 
Platform  Adaptor  Group  .'PAG) .  The  PAG  provides  a  transparent  subscriber 
interface  between  the  shipboard  JTIDS  terminal  and  existing  ships'  Combat 
Direction  Systems  (CDSs) . 

The  study  developed  a  Software  Quality  Assurance  (QA)  Plan  for  use  by 
NAVSEA  during  PAG  software  development.  The  QA  Plan  identifies  software 
QA  tasks  and  responsibilities  for  tasks  that  should  be  performed  during  the 
various  phases  of  PAG  software  development.  It  should  be  promulgated  for 
use  by  all  participants  in  the  PAG  development,  particularly  the  software 
developer  and  the  independant  VJ»V  agent. 

The  study  also  developed  a  structure  for  a  life-cycle-cost  model  to  be 
used  during  the  design  of  the  PAG.  The  model  structure  describes  life-cycle- 
cost  elements  to  be  included  in  a  future  cost  model.  It  is  recommended  that 
NAVSEA  complete  the  model,  including  detailed  cost  relationships,  as  a  com¬ 
puterized  tool  for  use  in  conducting  PAG  design  trade-off  studies. 
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ATTACHMENT:  SOFTWARE  QUALITY  ASSURANCE  PLAN  FOR  THE  JOINT 
TACTICAL  INFORMATION  DISTRIBUTION  SYSTEM 
PLATFORM  ADAPTOR  GROUP 

(submitted  under  separate  cover) 


CHAPTER  ONE 


INTRODUCTION 


Under  contract  N00I40-80-D-1052  with  the  Naval  Sea  Systems  Command 
(NAVSEA) ,  ARINC  Research  Corporation  was  tasked  to  conduct  a  study  in  support 
of  the  development  of  a  Platform  Adaptor  Group  (PAG)  for  the  Joint  Tactical 
Information  Distribution  System  (JTIDS) .  This  final  summary  report  describes 
the  results  of  the  contract  efforts  performed  for  NAVSEA. 

1.1  BACKGROUND 

The  Joint  Tactical  Information  Distribution  System  (JTIDS)  is  an  advanced 
information  distribution  system  that  provides  communications,  navigation, 
and  identification  (CNI)  capabilities  in  an  integrated  form  for  application 
to  military  and  air  defense  operations.  The  Naval  Sea  Systems  Command 
(NAVSEA)  is  responsible  for  shipboard  integration  of  this  system  into  U.S. 

Navy  ships  and  submarines,  including  the  development  of  a  PAG  to  provide  a 
transparent  subscriber  interface  between  the  JTIDS  terminal  and  existing 
ships'  Combat  Direction  Systems  (CDSs) .  The  PAG  will  contain  a  preprocessor 
providing  JTIDS  capability  to  communicate  and  exchange  data  with  other 
shipboard  systems  without  major  modifications  to  existing  shipboard  CDSs. 

1.2  OBJECTIVES 

The  purpose  of  this  study  was  to  provide  program  support  to  NAVSEA  in 
developing  the  PAG.  Since  a  major  portion  of  the  PAG  functions  will  be 
performed  by  computer  programs,  verification  and  validation  (V&V)  procedures 
are  required  for  the  PAG  software  and  associated  CDS  software  modifications. 

A  major  objective  of  this  study,  then,  was  to  provide  a  comprehensive  set 
of  software  V&V  procedures  to  be  used  during  PAG  development. 

In  addition,  techniques  will  be  needed  for  early  development  of  life¬ 
cycle-cost  (LCC)  estimates.  These  estimates  are  required  for  evaluating 
alternate  system  concepts  and  support  procedures,  detecting  cost  problems, 
and  providing  a  basis  for  formal  cos t-of-owner ship  considerations,  such  as 
logistics  support  and  preoperational  support  costs.  Thus  the  second  major 
objective  of  the  study  was  to  develop  a  structure  for  a  life-cycle-cost 
model  that  could  subsequently  be  expanded  to  a  full  life-cycle-cost  model 
for  use  in  conducting  trade-off  studies  during  PAG  design. 

1.3  REPORT  ORGANIZATION 

Chapter  Two  of  this  report  presents  a  description  of  the  technical 
approach  used  to  prepare  PAG  software  V&V  procedures  and  to  develop  a 
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structure  for  a  life-cycle-cost  model.  Chapter  Three  introduces  the  software 
VSV  procedures  and  describes  the  development  of  a  cost-model  structure  and 
the  life-cycle-cost  elements  to  be  included  in  a  future  model  for  the  conduct 
of  design  trade-offs.  Chapter  Four  presents  the  conclusions  and  recommenda¬ 
tions  resulting  from  the  study. 

The  V&V  procedures  to  be  used  during  PAG  software  development  are  pre¬ 
sented  in  a  separately  bound  attachment  to  this  report,  the  Software 
Quality  Assurance  Plan. 
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CHAPTER  TWO 


TECHNICAL  APPROACH 


Two  tasks  were  performed  to  achieve  the  desired  objectives:  (1)  develop 
V&V  procedures  for  controlling,  implementing,  testing,  and  reviewing  the 
PAG  software  as  it  is  developed;  and  (2)  formulate  a  structure  for  a  life¬ 
cycle-cost  model  to  be  used  during  PAG  design  in  the  conduct  of  trade-off 
studies.  The  technical  approaches  used  for  these  tasks  are  described  in 
Sections  2.1  and  2.2. 

2.1  TASK  1:  PREPARE  PAG  SOFTWARE  V&V  PROCEDURES 

During  the  software  development  process,  it  is  necessary  to  have  a  set 
of  procedures  for  reviewing  the  products  of  each  phase  of  the  software  develop¬ 
ment  process  to  ensure  that  these  products  meet  all  requirements.  This  pro¬ 
cess  is  referred  to  as  V&V.  Verification  is  the  iterative  process  of  deter¬ 
mining  whether  the  documents  produced  during  successive  steps  of  the  program 
development  process  satisfy  the  requirements  created  by  the  previous  steps. 
Validation  comprises  those  test  and  evaluation  activities  carried  out  to 
ensure  that  software  performance  is  consistent  with  design  documents. 

The  approach  used  to  develop  the  PAG  V&V  procedures  was  to  perform  the 
following  steps: 

1.  Identify  the  requirements  Cor  performing  PAG  software  development 

2.  Identify  the  responsibilities  of  the  performing  organizations 

3.  Define  potential  methods  for  performing  V&V 

4.  Identify  existing  methods  for  performing  V&V 

5.  Develop  a  cost-effective  set  of  methods  for  performing  V&V  based 
on  PAG  program  requirements 

6.  Coordinate  findings  with  Fleet  Combat  Direction  System  Support  Agency, 
San  Diego  (FCDSSASD)  and  NAVSEA 


7.  Document  results 


! 


The  following  documents  were  reviewed  in  the  performance  of  these 
steps: 

.  NAVSEA  Program  Management  plan 
.  Software  Management  Plan 
.  V&V  Management  Plan 

.  MIL-STD-1679,  Weapon  System  Software  Development 

.  JTIDS  PAG  Functional  System  Analysis 
.  JTIDS  Program  Plan,  Stage  I 
.  JTIDS  Configuration  Management  Plan 
.  JTIDS  Test  Concepts 
.  JTIDS  Quality  Assurance  Plan 

2.2  TASK  2:  DEVELOP  STRUCTURE  FOR  LIFE-CYCLE-COST  MODEL  FOR  PAG 

Task  2  does  not  develop  the  model  itself,  but  only  the  structure  of 
the  model.  Existing  in-house  models  that  apply  to  equipment  of  a  similar 
nature  were  analyzed.  The  first  is  the  life-cycle-cost  model  developed  to 
support  an  engineering  and  cost  analysis  for  the  JTIDS  Phase  II  terminals. 
It  is  documented  in  ARINC  Research  Publication  1731-01-2-1945*.  The 
second  is  a  model  used  to  conduct  a  cost  analysis  for  Army  User  Equipment 
of  the  NAVSTAR  Global  Positioning  System;  it  is  documented  in  ARINC 
Research  Publication  1172-02-3-1712**. 

Both  models  were  analyzed  to  determine  if  their  cost  categories  and 
cost  elements  corresponded  to  those  anticipated  for  the  PAG.  The  results 
of  this  analysis  were  used  to  develop  a  structure  for  the  PAG  life-cycle- 
cost  model.  Particular  attention  was  placed  on  software  costs,  since  a 
large  part  of  PAG  development  costs  will  be  incurred  in  the  development 
of  computer  programs. 


*Joint  Tactical  Information  Distribution  System  (JTIDS)  Phase  II: 
Terminals  Engineering  and  Cost  Analysis,  Volume  I,  25  June  1979. 

**Global  Positioning  System  Life-Cycle-Cost  Estimates  to  Support  Position¬ 
ing  and  Navigation  Cost  and  Operational  Effectiveness  Analysis, 

February  1978. 
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CHAPTER  THREE 


V&V  PROCEDURES  AND  LCC  MODEL  STRUCTURE 


3.1  PAG  SOFTWARE  V&V  PROCEDURES 

Software  development  is  a  structured,  organized  process  that  can  be 
divided  into  a  number  of  distinct  phases.  Each  phase  has  a  unique  purpose 
and  produces  a  specific  output.  The  output  of  each  phase  is  established  as 
a  baseline  for  development  of  the  succeeding  phase.  V&V  procedures  are 
necessary  for  each  phase  of  the  development  process  and  should  be  tailored 
to  the  specific  activity  of  each  phase. 

/ 

With  the  concurrence  of  NAVSEA  612,  the  V&V  procedures  to  be  used 
during  PAG  software  development  have  been  published  as  a  separately  bound 
document,  the  Software  Quality  Assurance  (QA)  Plan.  This  plan,  sub¬ 
mitted  as  an  attachment  to  this  report,  describes  all  of  the  tasks 
necessary  to  perform  V&V  on  the  PAG  software  during  the  following  phases: 

.  Requirements  definition 

.  Program  design 

.  Program  code,  compile,  and  debug 

.  Program  acceptance  test 

.  System  test 

3.2  LCC  MODEL  STRUCTURE 

This  section  describes  the  development  of  a  cost-model  structure  to  be 
used  during  PAG  design.  It  describes  the  life-cycle-cost  elements  to  be 
included  in  a  future  model  that  will  be  used  to  conduct  design  trade-offs. 

As  discussed  in  Section  2.2,  the  cost-model  structure  is  based  on  two  models 
previously  developed  by  ARINC  Research  Corporation.  The  cost  categories  and 
cost  elements  in  these  two  models  were  analyzed  to  determine  if  they  corre¬ 
sponded  to  anticipated  PAG  life-cycle  costs.  The  models  were  generally 
applicable,  with  the  exception  that  software  costs  were  not  adequately 
addressed.  Hence,  a  separate  category  for  software  costs  was  included  in 
the  PAG  cost  model's  structure. 
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As  a  result  of  this  analysis,  the  total  life-cycle  cost  was  divided 
into  four  categories: 

.  Research  and  development  (R&D) 

.  Investment 

.  Operating  and  support  (O&S) 

.  Software  development  and  support 

The  cost  elements  that  constitute  each  of  the  categories  are  defined  in  the 
following  sections. 

3.2.1  R&D  Costs 


RSD  costs  are  all  costs  required  to  develop  the  PAG  from  concept  design 
to  production.  Software  costs  are  excluded  since  they  have  been  assigned 
to  a  separate  category.  R&D  costs  include  the  following  elements: 

.  Development  engineering  -  engineering  cost  associated  with  the 
development  of  PAG  equipment,  including  all  prototypes  and  pro¬ 
duction  equipment 

.  Producibility  engineering  and  planning  -  cost  associated  with  equip¬ 
ment  producibility  studies  and  producibility  plans 

.  Tooling  -  cost  of  developing  the  unique  manufacturing  tooling 
necessary  to  produce  PAG  equipment 

.  Prototype  manufacturing  -  manufacturing  cost  associated  with  prototype 
equipment 

.  Data  -  cost  associated  with  obtaining  data  needed  to  develop  PAG 
equipment 

.  Test  and  evaluation  -  cost  of  performing  developmental  test  and 
evaluation  on  prototype  equipment 

.  Program  management  -  cost  of  program  management  for  PAG  development 

.  Training  -  cost  of  training  personnel  to  perform  developmental  test 
and  evaluation  or  other  developmental  tasks 

.  Facilities  -  cost  of  new  or  modified  facilities  to  develop  PAG  or 
conduct  developmental  test  and  evaluation 

.  Other  -  other  cost  not  falling  into  one  of  the  above  categories 
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3.2.2  Investment  Costs 


Investment  costs  are  those  costs  incurred  in  procuring  and  installing 
production  equipment  on  a  ship  or  other  platform  and  making  it  ready  for  use 
by  operational  forces.  Investment  costs  include  the  following: 

.  Equipment  purchase  -  cost  of  purchasing  production  PAG  equipment 

.  Initial  spares  purchase  -  cost  of  installation  and  checkout  (I&C) 
spares  and  the  initial  spares  inventory 

.  Installation  -  cost  of  installing  the  PAG  on  a  ship  or  other  platform, 
including  all  labor  and  material  required  for  the  installation 

.  Initial  training  -  cost  of  initial  training  of  personnel  to 
operate  each  delivered  unit 

.  Special  support  equipment  acquisition  -  cost  of  special  support 
equipment  required  to  support  PAG  production  and  installation 

.  First  destination  shipping  -  cost  of  shipping  each  production  equip¬ 
ment  to  the  initial  destination 

.  Engineering  change  proposals  (ECPs)  -  cost  of  developing  and  imple¬ 
menting  engineering  change  proposals  in  production  equipment 

.  Program  management  -  cost  of  program  management  for  PAG  production 

.  Documentation  -  cost  of  documentation  required  for  production 

.  Initial  inventory  management  -  cost  of  management  required  to 
establish  initial  inventory 

.  New  facilities  acquisition  -  cost  of  acquiring  or  modifying 
facilities  required  for  production 

3.2.3  O&S  Costs 


O&S  costs  are  those  costs  incurred  in  operating,  maintaining,  supplying, 
and  supporting  equipment  in  operational  use.  O&S  costs  include: 

.  Recurring  spares  -  cost  of  providing  the  necessary  spares  to  main¬ 
tain  and  operate  the  PAG 

.  Labor  -  costs  of  personnel  to  operate,  maintain,  supply,  and  support 
the  PAG 

.  Materials  -  costs  of  other  materials  and  services  (e.g.,  electricity) 
required  to  maintain  and  operate  the  PAG 

.  Support  equipment  operation  -  cost  of  maintaining  and  operating 
support  equipment  necessary  for  PAG  maintenance  and  operation 
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.  Recurring  training  -  cost  of  training  personnel  on  a  recurring  basis 


.  Inventory  holding  -  cost  of  holding  the  necessary  inventory  to  support 
the  PAG 

.  Facility  operation  -  cost  of  operating  and  maintaining  facilities 
required  for  PAG  support 

.  Program  management  -  cost  of  program  management  required  for  PAG 
support 

.  Recurring  inventory  management  -  cost  of  management  required  to 
manage  inventory 

.  Maintenance  transportation  -  shipping  charges  for  material  required 
to  maintain  the  PAG 

3.2.4  Software  Development  and  Support  Costs 

Software  development  and  support  costs  are  those  costs  of  developing  and 
maintaining  all  software  associated  with  the  PAG.  These  costs  include: 

.  Operational  software  development  -  cost  of  developing  the  operational 
PAG  software 

.  Test  software  development  -  cost  of  developing  the  test  software 
required  for  PAG  test  and  evaluation 

.  Support  software  development  -  cost  of  developing  any  support 

software  (e.g.,  compilers  and  models)  required  during  PAG  software 
development 

.  Operational  software  maintenance  -  cost  of  maintaining  operational 
software,  such  as  correcting  design  errors  and  incorporating  ECPs 

.  Test  software  maintenance  -  cost  of  maintaining  test  software  and 
developing  new  test  software  required  for  maintenance  of 
operational  software 

.  Support  software  maintenance  -  cost  of  maintaining  support 

software  and  developing  new  support  software  required  for  mainte¬ 
nance  of  operational  software 

3.2.5  Life-Cycle  Costs 

Table  3-1  summarizes  the  cost  categories  and  cost  elements  that  con¬ 
stitute  total  life-cycle  cost.  The  total  PAG  life-cycle  costs  are  the  sum 
of  R&D,  investment,  O&S,  and  software  development  and  support  costs.  The 
cost  for  each  of  these  categories  is  the  sum  of  the  individual  cost  elements 
within  that  category. 

Subsequent  tasks  are  planned  for  the  actual  development  of  the  model 
itself.  This  will  require  developing  cost  equations  for  each  cost  element 
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Table  3-1.  LIFE- 

CYCLE-COST  CATEGORIES  AND  COST  ELEMENTS 

Cost  Category 

Cost  Element 

Research  and  Develop¬ 
ment  (R&D) 

Development  Engineering 

Producibility  Engineering  and  Planning 

Tooling 

Prototype  Manufacturing 

Data 

Test  and  Evaluation 

Program  Management 

Training 

Facilities 

Other 

Investment 

Equipment  Purchase 

Initial  Spares  Purchase 

Installation 

Initial  Training 

Special  Support  Equipment  Acquisition 
First-Destination  Shipping 

Engineering  Change  Proposals 

Program  Management 

Documentation 

Initial  Inventory  Management 

New  Facilities  Acquisition 

Operating  and  Support 
(O&S) 

Recurring  Spares 

Labor 

Materials 

Support  Equipment  Operation 

Recurring  Training 

Inventory  Holding 

Facility  Operation 

Program  Management 

Recurring  Inventory  Management 

Maintenance  Transportation 

Software  Development 
and  Support 

Operational  Software  Development 

Test  Software  Development 

Support  Software  Development 

Operational  Software  Maintenance 

Test  Software  Maintenance 

Support  Software  Maintenance 
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in  the  model.  For  example,  the  cost  of  recurring  spares  is  a  function  of 
the  following  variables  at  a  minimum: 

.  Number  of  PAGs  in  operation 

.  Average  monthly  hours  of  operation 

.  Number  of  line  replaceable  units  (LRUs)  in  the  PAG 

.  Mean  time  between  failures  (MTBF)  for  each  LRU 

.  Average  cost  of  repairing/replacing  each  LRU 

.  MTBF  growth  for  each  LRU  (if  applicable) 

In  the  development  of  the  cost  of  recurring  spares,  these  and  possibly 
additional  variables  that  affect  the  total  cost  of  recurring  spares  for 
the  PAG  must  be  accounted  for  and  incorporated  in  the  cost  equation. 
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CHAPTER  POUR 


CONCLUSIONS  AND  RECOMMENDATIONS 


The  Software  QA  Plan,  submitted  as  an  attachment  to  this  report,  des¬ 
cribes  the  tasks  and  responsibilities  necessary  to  ensure  that  the  PAG  com¬ 
puter  programs  are  adequately  specified,  developed,  and  tested.  It  is 
recommended  that  this  plan  be  promulgated  by  NAVSEA  612  for  use  by  all 
participants  in  the  PAG  development.  In  particular,  it  should  be  referenced 
in  the  statements  of  work  for  both  the  PAG  software  developer  and  the  indepen¬ 
dent  VSV  agent,  so  that  each  organization  clearly  understands  its  role  and 
the  other's  role. 

The  cost-model  structure  contained  in  this  report  is  the  basis  for  the 
actual  development  of  the  cost  model  itself.  This  model  can  serve  a  useful 
purpose  during  PAG  design  as  a  tool  in  conducting  trade-off  studies.  It  is 
recommended  that  NAVSEA  complete  the  model  by  tasking  ARINC  Research 
Corporation  to  develop  the  detailed  cost  equations  needed  to  exercise  the 
model.  Because  of  the  complexity  of  the  model,  it  is  further  recommended 
that  a  computer  program  be  developed  to  facilitate  use  of  the  model. 
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SUMMARY 


The  Joint  Tactical  Information  Distribution  System  (JTIDS)  is  an  advanced 
information  distribution  system  that  provides  communications,  navigation, 
and  identification  (CNI)  capabilities  in  an  integrated  form  for  application 
to  military  and  air  defense  operations.  The  Naval  Sea  Systems  Command 
(NAVSEA)  is  responsible  for  shipboard  integration  of  this  system  into  USN 
ships  and  submarines,  including  the  development  of  a  Platform  Adaptor  Group 
(PAG)  to  provide  a  subscriber  interface  between  the  JTIDS  terminal  and  the 
existing  ships'  combat  direction  systems  (CDSs) .  This  study  for  NAVSEA 
developed  a  software  quality  assurance  (QA)  plan  for  use  by  NAVSEA  during 
PAG  software  development.  It  identifies  software  QA  tasks  and  responsibil¬ 
ities  for  those  tasks  that  should  be  performed  during  the  following  phases 
of  software  development:  requirements  definition,  program  design,  program 
code,  compiling  and  debugging,  acceptance  test,  and  system  test. 
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CHAPTER  ONE 


INTRODUCTION 


1 . 1  PURPOSE 

The  purpose  of  the  Platform  Adaptor  Group  (PAG)  software  quality  assur¬ 
ance  (QA)  plan  is  to  identify  the  specific  responsibilities  and  actions 
required  of  organizations  participating  in  the  development  and  test  of  PAG 
software.  Emphasis  will  be  placed  primarily  on  the  responsibilities  and 
actions  required  of  the  PAG  software  developer,  although  other  organizations 
such  as  FCDSSASD ,  NAVSEA,  and  the  verification  and  validation  (VS V)  agent 
will  be  covered  as  well.  The  primary  objective  of  these  procedures  is  to 
define  a  plan  for  software  QA  that  will  support  the  software  development 
approach  already  established  by  FCDSSASD.  Identification  of  the  specific 
plan  in  advance  of  actual  software  development  will  establish  a  baseline 
against  which  participating  organizations  can  plan  their  efforts. 


1 . 2  BACKGROUND 

Prior  to  the  establishment  of  the  Joint  Tactical  Information  Distribu¬ 
tion  System  (JTIDS)  program,  the  Air  Force  and  the  Navy  were  independently 
planning  and  developing  tactical  command  and  control  systems  based  on  time- 
division  multiple-access  (TDMA)  signal  technology.  The  Navy's  programs 
were  the  Integrated  Tactical  Naviqation  System  (ITNS)  and  the  Integrated 
Tactical  Air  Control  System  (ITACS).  The  former  developed  and  tested  a 
TDMA  system  that  provided  relative  position  determination  to  system  partic¬ 
ipants,  and  the  latter  demonstrated  the  use  of  common  equipment  for  commu¬ 
nications,  navigation,  and  identification  and  was  able  to  transmit  secure, 
jam-protected  digital  information.  The  Air  Force's  program  was  SEEK  BUS, 
which  developed  a  TDMA  secure,  jam-protected  digital  information  system  that 
emphasized  connectivity  between  subscribers. 

Because  of  the  similarities,  the  programs  were  merged  in  1974  to  form 
the  JTIDS  program.  A  Joint  Program  Office  (JPO)  was  formed,  and  the  Air 
Force  was  designated  as  the  lead  service,  with  the  Air  Force  Systems  Command 
(AFSC)  as  the  implementing  command.  The  JTIDS  JPO  has  a  joint  program  man¬ 
ager  who  is  responsible  for  planning,  organizing,  coordinating,  controlling, 
and  directing  the  definition,  development,  production,  procurement,  and 
financial  management  of  the  program.  He  is  assisted  by  a  deputy  program 
manager  from  each  service.  The  deputy  program  manager  (Navy)  is  the  single 
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point  of  contact  between  the  Navy  and  the  JTIDS  joint  program  manager. 
Information  on  the  Navy  JTIDS  organization  is  contained  in  Section  2.3. 


1.3  JTIDS  PROGRAM  DESCRIPTION 

JTIDS  provides  CNI  capabilities  through  its  ability  to  distribute  infor¬ 
mation  at  high  rates,  encrypted  to  provide  security  and  reliability  in  hos¬ 
tile  environments.  Security  and  jamming  resistance  are  achieved  through  the 
use  of  pseudorandom  signal-processing  techniques. 

The  system  provides  the  capability  of  interconnecting  scattered  sources 
of  surveillance,  support,  and  intelligence  information,  weapons  controllers, 
weapons  systems,  and  decision-making  commanders  —  with  selectable  levels  of 
connectivity  among  these  elements  —  so  that  the  tactical  commander  may 
structure  and  restructure  his  forces  on  a  continuing  real-time  basis  as  he 
views  the  combat  situation.  It  provides  mobile  surface  and  airborne  force 
elements  with  a  relative  navigation  capability  within  a  common  position- 
reference  grid  and  with  an  intrinsic  identification  capability  through  the 
dissemination  of  crypto-secure  position,  velocity,  and  identity  information 
concerning  both  friendly  and  hostile  force  elements. 

Three  time-division  multiple-access  signal  architectures  have  been 
developed.  They  are  all  based  on  a  single  communications  circuit  that  simul¬ 
taneously  services  multiple  users.  The  basic  architecture  is  referred  to  as 
TDMA.  Advanced  TDMA  (ATDMA)  is  an  extension  of  TDMA  to  provide  capability 
for  greater  information  throughput,  and  distributed  TDMA  ( DTDMA )  uses  the 
same  transmission  symbols  as  TDMA  and  ATDMA  but  also  disperses  those  symbols 
pseudorandomly  in  time.  Each  participant  in  the  TDMA  or  ATDMA  network  is 
equipped  with  a  synchronized  clock  and  is  assigned  a  sufficient  number  of 
time  slots  to  accommodate  the  number  of  messages  likely  to  be  required  by 
his  mission.  During  his  assigned  time  slot,  each  user  broadcasts  data  into 
a  commonly  accessible  communications  data  stream.  DTDMA  network  participants 
are  not  assigned  time  slots  but  transmit  pseudorandomly.  All  network  parti¬ 
cipants  can  extract  information  they  require  by  continuously  monitoring  and 
sampling  the  data  transmission . 

Three  classes  of  terminal  equipment  are  provided,  and  the  characteris¬ 
tics  of  each  class  are  defined  to  satisfy  the  needs  and  capabilities  of  the 
various  types  of  user  systems,  particularly  with  respect  to  size  and  weight. 
Class  1  terminals  provide  the  highest  level  of  capability  for  use  by  large- 
scale  airborne  and  surface  command  and  control  systems.  Class  2  terminals 
provide  similar  capabilities  but  have  a  lower  level  of  RF  power  output  and 
may  have  less  capability  for  information  throughput.  The  smaller,  lighter 
Class  2  terminals  are  intended  for  use  by  force  elements  such  as  fighter 
aircraft  and  small  ships.  Class  3  terminals  are  more  compact,  lower-cost 
versions  for  applications  such  as  man-packs  and  missile  guidance.  Modular 
additions  to  Class  2  terminals  to  obtain  Class  1  terminal  capability  and 
"command  terminal"  capability  for  use  on  aircraft  carriers  and  E-2  aircraft 
have  also  been  identified. 


JTIDS  will  be  implemented  in  a  two-stage  approach  to  ensure  that  inter¬ 
operability  and  compatibility  requirements  are  met  completely.  Stage  I  will 
seek  to  minimize  the  impact  of  the  JTIDS  terminal  on  existing  combat  direc¬ 
tion  systems  (CDSs)  by  using  the  PAG  as  an  interfacing  device  to  communicate 
between  the  CDS  and  the  JTIDS  terminal.  The  PAG  will  allow  JTIDS-equipped 
platforms  to  communicate  with  each  other  via  Link  16  and  still  maintain  a 
communication  capability  via  existing  digital  links,  Link  4A  and  Link  11. 

Stage  I  will  provide  simultaneous  operation  of  the  following: 

•  Existing  Link  4A  and  Link  11  capabilities 

•  Jam-resistant,  secure  data  exchange  via  JTIDS  using  existing  TADIL-A, 
TADIL-C,  and  selected  TADIL-J  message  standards  in  the  existing  CDS 

•  Rel-Nav  capability 

•  Jam-resistant,  secure,  digital  voice 

•  JTIDS  relay  capability 

•  Interoperability  with  other  services  on  voice  channels  and  precise 
position  location  identification  (PPLI)  data  channels 


Stage  II  will  provide  full  JTIDS  capabilities  that  will  totally  integrate 
the  J-series  messages  into  the  CDS. 


1 . 4  SCOPE 

This  software  QA  plan  defines  all  software  QA  activity  that  is  required 
during  the  development  of  the  PAG  software,  beginning  with  the  establishment 
of  the  PAG  functional  baseline  and  ending  with  the  establishment  of  the 
operational  baseline.  It  includes  the  following  activities: 

•  Program  requirements  definition 

•  Program  design 

•  Program  code  and  debug 

•  Subprogram  and  module  test  (SP/MT) 

•  Function  test  (FT) 

•  Program  acceptance  test  (PAT) 

•  System  integration  test  (SIT) 

•  Navy  interoperability  test  (NIT) 

The  goal  of  software  QA  is  to  provide  a  structured  approach  to  software 
development  that  r.-su  1l.s  in  a  high-quality  product.  It  includes  all  activ¬ 
ities  designed  to  achieve  this  goal,  particularly  V&V  and  configuration- 
management  (CM)  activities. 
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This  plan  applies  to  JTIDS  Stage  I  only.  The  software  QA  plan  for 
JTIDS  Stage  II  will  be  defined  at  a  later  date.  The  PAG  requirements  dis¬ 
cussed  herein  are  those  intended  to  interface  a  Class  1  terminal  with  a 
shipboard  CDS.  Class  2  terminal  PAG  requirements  may  be  similar,  but  these 
are  not  addressed  in  this  report. 


1.5  APPLICABLE  DOCUMENTS 

The  documents  listed  below  contain  guidance  and  direction  necessary  to 
the  performance  of  software  QA  tasks  for  the  PAG  computer  programs. 

•  DOD-STD-480A,  Configuration  Control  -  Engineering  Changes,  Devia¬ 
tions  and  Waivers,  dated  29  December  1978 

•  MIL-STD-483,  Configuration  Management  Practices  for  Systems,  Eguip- 
ment ,  Munitions  and  Computer  Programs ,  dated  21  March  1979 

•  MIL-STD-1679 ,  Military  Standard  Weapon  System  Software  Development , 
dated  1  December  1978 

•  SECNAVINST  3560.1,  Department  of  the  Navy  Tactical  Digital  Systems 
Documentation  Standards,  dated  8  August  1974 

•  OPNAVINST  3960.10,  Test  and  Evaluation,  dated  22  October  1975 

•  FCDSSASD  JTIDS  Program  Plan,  Stage  I,  M(P)-511S,  dated  31  October 
1980 

•  FCDSSASD  JTIDS  Test  Concepts,  IR(P)-2427,  dated  31  October  1980 

•  FCDSSASD  JTIDS  Configuration  Management  Plan,  M(P)-5132,  dated 
23  December  1980 

•  FCDSSASD  JTIDS  Quality  Assurance  Plan,  M(P)-5133,  dated  23  December 
1980 


FCDSSA  Technical  Report  on  JTIDS  PAG  System  Functional  Analysis 
FR(P)-3162,  dated  31  October  1980 


CHAPTER  TWO 


QA  PLAN  OVERVIEW 


2.1  JTIDS  FUNCTIONAL  DESCRIPTION 

A  top-level  diagram  illustrating  the  integration  of  JTIDS  shipboard 
equipment  with  a  shipboard  CDS  is  shown  in  Figure  2-1.  The  essential  ele¬ 
ments  of  the  diagram  are  as  follows: 

•  A  TADIL-J  terminal  group  providing  secure,  jam-resistant,  high- 
capacity  information  distribution  in  accordance  with  systems 
specification  DCB  76S0000 

•  A  Platform  Adaptor  Group  (PAG)  consisting  of  a  PAG  computer  suite, 
control  and  display  terminals,  and  support  and  test  equipment  that 
provides  for  the  integration  of  JTIDS,  TADIL-A,  and  TADIL-C  terminal 
classes  and  signal  architectures  with  a  CDS 

•  A  voice  control  group  providing  automated  switching  of  analog  and 
digital  voice  communications  as  dictated  by  an  external  subscriber 
network  controlled  by  the  PAG 

•  A  TADIL-A  terminal  group  providing  for  subscriber  information 
exchange  via  M-series  messages  in  accordance  with  OPSPEC  411.1 

•  A  TADIL-C  terminal  group  providing  for  subscriber  information 
exchange  via  V/R-series  messages  in  accordance  with  OPSPEC  404 

•  A  CV/CVN  combat  direction  system  (CDS)  group  consisting  of  NTDS 
Model  4.0  equipment  suites  and  computer  programs  that  support  the 
CV/CVN  mission 

•  A  navigation  group  consisting  of  a  Ship's  Inertial  Navigation 
System  (SINS)  computer  suite  and  associated  manual-navigation-input 
sources 

In  addition,  a  group  of  switches  (SW-1/1A,  SW-2/2A,  SW-3/3A)  has  been 
provided  for  interconnection  of  the  TADIL-A,  TADIL-C  and  navigation  groups 
with  the  PAG  and  CDS  for  casualty-mode  operations. 


2.2  PAG  DESCRIPTION 

The  PAG  serves  as  the  interfacing  unit  to  allow  the  CDS  to  communicate 
with  the  JTIDS  terminal.  The  PAG  has  been  designed  to  allow  for  a  minimal 
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Figure  2-1.  JTIDS/CDS  TOP  LEVEL  DIAGRAM 


impact  on  the  existing  shipboard  CDS.  Figure  2-1  shows  the  equipment 
comprising  the  PAG:  a  computer  suite,  control  and  display  terminals,  and 
support  and  test  equipment.  Functional  allocations  to  specific  hardware  com¬ 
ponents  or  between  hardware  and  software  have  not  yet  been  made;  the  follow¬ 
ing  sections  describe  functions  that  will  be  contained  within  the  PAG  func¬ 
tional  elements. 

2.2.1  PAG  Computer  Suite 

The  PAG  computer  suite  will  contain  all  software  programs  necessary 
to  the  following  functions: 

•  Executive  control  functions  -  software  that  controls  program  execu¬ 
tion,  which  includes  such  functions  as  initialization,  interrupt 
processing,  scheduling,  dispatching,  I/O  processing,  executive- 
service-request  processing,  and  error  processing 

•  TADIL-J  interface  function  -  software  that  implements  protocol  and 

I/O  techniques  to  allow  communication  and  exchange  of  J-series 

messages  via  the  TADIL-J  terminal  group 

•  TADIL-C  interface  function  -  software  that  implements  protocol  and 

I/O  techniques  to  allow  communication  and  exchange  of  V/R-series 

messages  via  the  TADIL-C  terminal  group 

•  TADIL-A  interface  function  -  software  that  implements  protocol  and 

I/O  techniques  to  allow  communication  and  exchange  of  M-series 

messages  via  the  TADIL-A  terminal  group 

•  Navigation  function  -  software  that  implements  the  capability  to 
interface  REL-NAV  data  from  the  TADIL-J  terminal  group  with  own- 
ship's  navigation  group  and  CDS  group 

•  Man-machine  interface  function  -  software  that  implements  the 
capability  to  monitor  on-line  status  of  net  operations,  system 
software,  and  system  hardware  as  required  to  maintain  specified 
levels  of  operational  capability. 

•  Voice-control  function  -  software  that  implements  the  capability  to 
control  voice  communications  between  JTIDS  units  and  ownship's 
voice- transmission  equipments 

•  On-line  test  function  -  software  that  implements  the  capability  to 
perform  on-line  nondestructive  tests  designed  to  provide  fault 
detection  and  isolation  of  hardware  and/or  software  malfunctions 

•  Data  extraction  and  reduction  function  -  software  that  implements 
the  capability  to  extract  and  record  selected  data  in  real-time  for 
both  on-line  and  off-line  reduction  in  support  of  system  test  and 
evaluation  as  well  as  post-mission  analysis 

•  Netway  -  software  that  permits  the  simultaneous  operation  of  TADIL-A 
and  JTIDS  nets  and  the  exchange  of  information  between  subscribers 
on  both  nets 
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Biway  -  software  that  permits  the  simultaneous  operation  of  Link-4A 
and  JTIDS  nets. 


2.2.2  PAG  Control  and  Display  Terminals 

One  or  more  PAG  control  and  display  terminals  of  the  keyboard-input/ 
CRT-output  type  will  be  required  to  permit  exchange  between  the  operator (s) 
and  the  PAG  for  system  control  and  operation.  The  terminal(s)  will  allow  the 
operator (s)  to  initialize  and  control  system  operation,  monitor  and  control 
net  activity,  control  voice-mode  operation,  perform  on-line  tests,  perform 
data  extraction  and  reduction,  and  initiate  casualty-mode  operations. 

2.2.3  PAG  Support  and  Test  Equipment 

Exact  specifications  for  PAG  support  and  test  equipment  have  not  been 
determined.  It  is  anticipated  that  a  magnetic-tape  unit  will  be  required 
for  program  loading,  and  on-line  data  extraction  and  reduction  for  system 
test,  evaluation,  and  post-mission  analysis. 


2 . 3  SOFTWARE  DEVELOPMENT 

The  software  development  approach  that  will  be  used  by  FCDSSASD  for 
PAG  software  is  shown  in  Figure  2-2.  It  consists  of  a  series  of  distinct, 
well-defined  phases.  Each  phase  produces  an  output  that  serves  as  the 
input  for  the  next  phase.  QA  activity  takes  place  during  each  phase  and 
assists  the  program  manager  to  determine  whether  the  next  phase  can  begin. 

As  an  aid  in  providing  an  orderly,  controlled  transition  from  one 
phase  to  the  next,  baselines  are  defined  and  established  at  the  conclusion 
of  each  phase  and  serve  as  points  of  departure  for  the  next  phase.  A  base¬ 
line  consists  of  formally  approved  and  configuration-controlled  technical 
documentation  that  defines  the  output  of  each  phase.  Figure  2-3  shows  the 
different  baselines  that  FCDSSA  will  establish  during  PAG  software  devel¬ 
opment.  Table  2-1  specifies  when  each  baseline  is  established  and  which 
defining  documentation  is  approved  and  placed  under  configuration  manage¬ 
ment  for  each  baseline.  Table  2-2  defines  the  phases  of  the  software- 
development  process  that  occur  between  baselines.  Chapter  Three  gives 
specific  V&V  procedures  for  each  phase  listed  in  Table  2-2. 
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iqure  2-3.  STAGE  I  JTIDS  SOFTWARE  BASELINES 


Table  2-1. 


BASELINE  ESTABLISHMENT  AND  DOCUMENTATION 


Baseline 

When  Established 

Approved  Defining 
Documentation 

Functional  (ABL) 

Prior  to  start  of  full- 
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2.4  SUPPORTING  ORGANIZATIONS 


JTIDS  responsibilities  within  the  Navy  are  assigned  as  directed  in 
OPNAVINST  C3510.13:  The  Command  and  Control  Directorate,  Chief  of  Naval 
Operations  (OP-094) ,  is  the  Navy  JTIDS  program  and  resource  sponsor;  the 
Office  of  Antisubmarine  Warfare,  Chief  of  Naval  Operations  (OP-95) ,  is  the 
Navy  JTIDS  mission  sponsor;  and  Chief  of  Naval  Material  (CNM)  is  the  execu¬ 
tive  agent  for  Navy  JTIDS  development.  As  executive  agent,  Chief  of  Naval 
Material  issued  a  letter  (Serial  031/DJS  of  27  August  1975)  defining  the 
responsibilities  of  the  commands  to  which  JTIDS  development  activities  are 
assigned  in  OPNAVINST  C3510.13. 

2.4.1  Naval  Electronics  Systems  Command  (NAVELEX) 

OPNAVINST  C3510.13  assigned  NAVELEX  the  responsibilities  of  system 
engineer  for  JTIDS.  The  CNM  letter  letter  designated  NAVELEX  as  the  prin¬ 
cipal  development  activity  for  Project  XCC-24,  JTIDS  architecture  and  non¬ 
avionics,  and  as  the  systems  command  responsible  for  supporting  the  Joint 
Program  Office. 

2.4.2  Naval  Sea  Systems  Command  (NAVSEA) 

NAVSEA  is  responsible  for  shipboard  integration  of  all  systems  into 
USN  ships  and  submarines.  OPNAVINST  C3510.13  assigned  NAVSEA  the  following 
responsibi 1 ities : 

•  Maintenance  and  enhancement,  during  and  following  JTIDS  implementa¬ 
tion,  of  lnteroperabil ity  among  joint  and  allied  systems  achieved 

by  the  tactical  air  control  systems  and  tactical  air  defense  systems 
(TACK  and  TADS)  interface  efforts  using  TADIL-A  (Link  11) 

! 

•  H  .using  the  mstal  1  .it  ion  of  JTIDS  in  specific  units  so  as  to 
ensure  minimum  interference  and  provide  maximum  operational  capa- 
bi  lit'/  to  t  h>  •  licet  commando: 

•  :  :■  ;  nr  at.  ion  no.  i  i  f  l  ■  at  ions  to  existing  •  -ombat-di reot  ion  systems 

software  m  •>*  lin.it  ion  with  JTIDS  terminal  software  to  maximize 
ef  feet  1  veriess  it  i  mnimizi  .oats  for  each  system 

•  I.it'VL  lot  mci.t.  an  i  implementation  <t  integrated  logistic  support  for 
NAVSEA-  -ogr.  i /.int  •  :u  i  pment  so  t  ha*  at:  effective  reliability  and 
mair.t  .u  r.abi  I  ;  ty  f  rograr  will  be  it  pi  led  f  t  on  initial  design  through 
di  •.'■  1  .ipment  ,  eng  l  ne.-rin  i ,  test  md  evaluation  (Tf»E)  ,  production, 

l :  1  f  a  1  1  at  i  on ,  • .[  .  t  it  in.,  md  suppot  t  phases 

•  I  {.  nt  if  ic  it  ion  t  >.-s(  >ut  •.  i  s  :  i  installation  and  life-cycle  support 

•  . i  JVIP.c  ...  ,  ]  •,  ...  possitd. 

Th>-  i 'NM  Its  i  mat.  1  NAY.d.A  i  -  .  ;  t  incip.il  -level  ament  activity  for 
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•  Coordination  of  all  internal  NAVSEASYSCOM  JTIDS  efforts,  including 
planning  and  liaison  with  all  ship  program  management  and  ship  logis 
tic  management  offices 

•  Development  of  shipboard  installation  and  integration  plans  in 
support  of  overall  JTIDS  program  goals 

•  Assistance  to  NAVELEX  in  an  integrated  plan  to  incorporate  JTIDS 
capabilities  for  surface  and  subsurface  use  other  than  directly 
interfacing  into  NTDS 

•  Development  of  a  NAVSEASYSCOM  budget  as  a  separate  JTIDS  project 
number  for  shipboard  TDS  integration 

•  Maintenance  of  a  NAVSEA  project  office  to  carry  out  these  responsi¬ 
bilities  under  the  Navy  program  manager. 

2.4.3  Fleet  Combat  Direction  System  Support  Activity  (FCDSSA) 

FCDSSAs  at  San  Diego  and  Dam  Neck  are  responsible  for  assisting  NAVSEA 
by  developing  the  necessary  software  for  interfacing  JTIDS  and  NTDS  and 
other  combat-direction  systems  installed  in  Navy  ships.  FCDSSA,  San  Diego, 
is  responsible  for  developing  interface  processor  software  and  modifications 
to  CDS  operational  programs  for  CV/CVN,  LCC,  and  LHA  ships.  FCDSSA,  Dam  Neck 
is  responsible  for  developing  modifications  to  interface  processor  software 
and  CDS  operational  programs  for  CG,  CGN ,  DDG ,  DD,  and  FFG  ships. 

2.4.4  Operational  Test  and  Evaluation  Force  (OPTEVFOR) 

OPTEVFOR  is  the  Navy's  independent  testing  agency  for  planning,  con¬ 
ducting,  and  reporting  the  operational  test  and  evaluation  of  equipment  and 
systems  being  procured  for  the  Navy.  OPTEVFOR  also  monitors  all  pertinent 
phases  of  developmental  test  and  evaluation  (T&E) .  OPTEVFOR  is  tasked  by 
the  deputy  program  manager  (Navy)  in  support  of  the  Joint  Program  Test  and 
Evaluation  Master  Plan,  and  by  the  Navy  program  manager  in  support  of  the 
Navy  Test  and  Evaluation  Master  Plan. 

2.4.5  I independent  Vi»V  Agent 

The  inde;  endent  Vf.V  agent  is  responsible  for  performing  V&V  for  the 
PAG  software  modifications  during  full-scale  development.  The  independent 
VS V  agent  should  be  a  government  agency  or  contractor  not  associated  with 
any  other  part  of  the  PAG  software  development ,  and  he  has  the  following 

r*  'Spons  l  b  1  it  ns: 


Ensuring  the  interoperability  of  the  PAG  software  with  the  JTIDS 
class  1  terminal  and  the  shipboard  CDS 

Performing  V.sV  on  all  support  programs,  such  as  simulators  and 
1  C  i  •  ■  -  r  :j.  t m  i  i  •  •  iu  •(.  i  s 

Reviewing  ill  lot.  umentation  for  completeness,  conformance  to  mili¬ 
tary  st  mdur-is,  and  consistency  with  other  program  documentation 
arm  *  In  computer  [loqram  i  aekaqe  itself 


•  Monitoring  test  conduct 

•  Independently  assessing  system  status 

•  Verifying  that  test  plans,  specifications,  and  procedures  adequately 
test  the  system 

•  Independently  analyzing  test  results 

■  Developing  the  PAT  test  documentation 

•  Directing  the  conduct  of  the  PAT 


2.5  VERIFICATION  AND  VALIDATION 

Verification  is  the  iterative  process  of  determining  whether  the  docu¬ 
ments  produced  during  successive  steps  of  a  program-development  process 
fulfill  the  requirements  levied  by  the  previous  steps.  The  purpose  of  veri¬ 
fication  is  to  demonstrate  the  consistency,  completeness,  and  correctness 
of  the  software  at  each  stage  of  development. 

Verification  activities  can  take  place  during  each  stage  of  the  devel¬ 
opment  process:  requirements  definition,  design,  code,  and  test.  The 
extent  of  verification  varies  from  program  to  program.  Pertinent  factors 
that  affect  the  amount  of  verification  performed  include  program  size,  avail¬ 
able  budget,  and  criticality  of  the  software. 

Validation  consists  of  test  and  evaluation  activities  that  are  per¬ 
formed  to  ensure  that  software  performance  is  consistent  with  specification 
requirements.  Validation  also  determines  that  the  software  properly  meets 
the  user's  needs  and  requirements,  a  task  that  includes  an  evaluation  of  the 
software  requirements  themselves.  Like  verification,  the  scope  of  valida¬ 
tion  can  vary  from  program  to  program. 

Chapter  Three  of  this  report  contains  verification  and  validation  pro¬ 
cedures  that  are  judged  to  be  cost-effective  for  the  PAG  software  develop¬ 
ment.  It  is  not  the  intent  of  this  report  to  define  specific  test  plans, 
methods,  specifications,  or  procedures,  but  rather  to  provide  a  set  of  pro¬ 
cedures  for  defining  the  verification  and  validation  activity  that  must  take 
place. 
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CHAPTER  THREE 


SOFTWARE  QA  APPROACH  BY  PHASE 


This  chapter  contains  a  detailed  discussxon  of  the  approach  for  per¬ 
forming  software  QA  during  each  phase  of  software  development.  QA  activity 
is  defined  separately  for  each  phase.  The1'.,  phases  are  related  to  the  soft¬ 
ware  baselines  established  during  software  development,  as  shown  in  Table 
2-2.  Detailed  QA  approaches  for  use  with  all  phases  are  given  in  the 
following  sections  of  this  chapter: 

•  Requirements  definition 

•  Program  design 

•  Program  code  and  debug 

•  Program  acceptance  test 

•  System  test 

•  Maintenance 

Each  section  describes  the  major  activity  that  takes  place  during 
each  phase,  starting  with  the  beginning  baseline  for  each  phase  and  ending 
with  the  establishment  of  the  baseline  for  the  next  phase,  or,  in  the 
case  of  the  maintenance  phase,  the  end  of  the  system  life  cycle.  Once  the 
major  activity  of  each  phase  is  described,  there  follows  an  account  of  the 
specific  QA  activity  for  each  phase,  the  QA  tasks  to  be  performed,  and  the 
responsibilities  for  performing  QA  tasks.  To  provide  an  orderly  discus¬ 
sion,  the  QA  tasks  to  be  performed  in  each  phase  are  placed  in  the  follow¬ 
ing  general  categories: 

•  Documentation  review 

•  Design  reviews  and  audits 

•  Configuration  management 

•  Test  and  evaluation 

•  Status  reports 


3.1  REQUIREMENTS- DEFINITION  PHASE 

The  roquirements-di  •  f  i  m  1. 1  :  hi.se  begins  with  the  establishment  of  the 

functional  baseline  an  1  ends  with  t  Ik-  •  st ai  1 j  shmont  of  the  a 1 located-Part  I 


baseline.  The  functional  baseline  is  established  in  accordance  with  the 
top-level  operational  requirements  (TOR)  and  the  interface  design  require¬ 
ments  (IDR)  after  a  system  design  review  (SDR)  has  been  held  to  review 
preliminary  versions  of  these  documents.  Once  all  comments  from  the  SDR 
have  been  resolved,  the  TOR  and  IDR  are  approved  and  placed  under  CM  con¬ 
trol.  This  represents  the  departure  point  for  the  software  requirements- 
definition  phase. 

Figure  2-2  shows  the  activity  that  occurs  during  the  requirements- 
definition  phase.  The  object  of  this  phase  is  to  develop  the  performance 
requirements  for  the  overall  system,  the  hardware,  and  the  computer  programs 
that  support  the  mission  requirements  defined  in  the  functional  baseline, 
i.e.,  the  TOR  and  IDR.  The  primary  system  and  computer  program  documenta¬ 
tion  produced  durinq  this  phase  is  as  follows: 

•  System  operational  specification  (SOS) 

•  System  operational  design  (SOD) 

•  PAG  program  performance  specification  (PPS) 

•  PAG  interface  design  specifications  (IDSs) 

The  SOS  describes  in  detail  guidelines  for  implementing  the  mission 
requirements  contained  in  the  TOR  and  provides  program  performance  guidance 
and  equipment  constraints.  The  SOD  is  then  prepared  to  provide  a  technical 
planning  document  that  defines  the  hardware  environment,  the  software 
functional  operations,  and  the  interfaces  between  the  PAG  and  the  JTIDS 
terminal  and  existing  CDS. 

Detailed  software  performance  requirements  are  now  prepared.  The  IDSs 
specify  the  software  interfaces  between  the  PAG  and  external  systems.  Five 
IDSs  involving  the  PAG  are  anticipated.  These  will  specify  the  software 
interface  between  the  PAG  and  the  following  systems  and  equipment: 

•  JTIDS  terminal 

•  Link  4A  data  terminal  set 

•  Link  11  data  terminal  set 

•  Navigation  system 

•  CDS 

The  PPS  contains  all  operational  and  technical  requirements  necessary  to 
design,  test,  and  maintain  the  PAG  computer  program.  The  development  of 
the  PPS  and  that  of  the  IDS  normally  t  ike  i  la  e  simultaneously. 

Another  activity  normally  •  I  iur  ir:o  this  phase  is  planning  to 

define  the  requirements  for  at,-,  i .  i  *  -Is  » hat  will  be  required  in  sub¬ 
sequent  phases.  This  |  I  ■  i  r . : ,  1 : .  ■  i  1  :  ■  u  1  •,  i  development  process  to 

ensure  that  adequate  le  id  t  in»  i  t 1  w>-  i  •  i-.-.wn,  implement,  test,  and 

certify  these  tools  be!  or.  it  *  •  .  t  >  :  . 


At  the  conclusion  of  the  requirements-definition  phase,  the  SOS,  SOD, 
PPS ,  and  IDSs  are  all  approved  by  the  Navy  and  placed  under  CM  control. 

These  documents  then  constitute  the  allocated-Part  I  baseline  for  the  PAG 
computer  programs  and  constitute  the  stepping-off  point  for  the  next  phase 
-  program  design.  The  following  sections  contain  detailed  QA  tasks  to  be 
performed  during  the  requirements-definition  phase. 

3.1.1  Documentation  Review 

As  shown  in  Figure  2-2,  four  primary  documents  are  developed  during 
the  requirements-definition  phase:  the  SOS,  SOD,  PPS,  and  IDSs.  While  all 
four  are  formally  reviewed  at  a  preliminary  design  review  (PDR) ,  the  purpose 
of  this  document  review  is  to  evaluate  their  acceptability  prior  to  the  PDR. 
Three  types  of  document  review  will  be  conducted  for  the  SOS,  SOD,  PPS,  and 
IDSs,  as  follows: 

•  Format  review 

•  Content  review 

•  Comprehensive  review 

3. 1.1.1  Format  Review 

The  purpose  of  the  format  review  is  to  determine  that  the  expression 
and  format  meet  all  applicable  specifications.  Each  document  is  reviewed 
for  format  against  the  requirements  contained  in  the  following  instructions 
and  data  item  descriptions  (DIDs) : 

•  SOS  -  Secretary  of  the  Navy  Instruction  (SECNAVINST)  3560.1 

•  SOD  -  SECNAVINST  3560.1 

•  PPS  -  DID  DI-E-2136 

■  IDSs  -  DID  DI-2] 35 

3 . 1 . 1 . 2  Content  Review 

The  purpose  of  the  content  review  is  to  determine  that  the  operational 
and  technical  content  of  the  document  is  operationally  and  technically 
feasible.  The  higher- level  documents,  the  SOS,  SOD,  and  PPS,  are  written 
in  terms  of  fleet  operations  and  missions,  and  are  reviewed  for  operational 
feasibility.  The  lower-level  documents,  the  PPS  and  IDSs,  are  written  in 
terms  of  software  architecture  and  interfaces,  and  are  reviewed  for  tech¬ 
nical  feasibility.  Note  that  the  PPS  is  a  combination  of  the  operational 
and  the  technical  and  is  reviewed  both  ways.  The  operational  and  technical 
content  review  are  conducted  by  specialists  in  their  respective  areas.  An 
important  factor  in  this  review  is  that  the  "shotgun"  approach,  where  each 
expert  reviews  the  entire  document,  is  to  be  avoided.  A  document  should 
be  divided  into  various  areas,  with  reviewers  assigned  to  review  specific 
areas  in  which  they  are  expert.  This  ensures  that  the  entire  document 
is  reviewed,  and  it  minimizes  the  duplication  of  effort. 


3. 1.1. 3  Comprehensive  Review 


The  purpose  of  the  comprehensive  review  is  to  determine  overall  docu¬ 
ment  comprehensiveness.  Reviewers  assigned  to  conduct  a  comprehensive 
review  will  accomplish  the  following: 

•  Evaluate  document  content  structure 

•  Determine  if  boundaries  are  properly  described 

•  Ensure  traceability  of  requirements  to  higher-level  documents 

•  Determine  if  the  document  is  consistent  within  itself 

•  Ensure  that  it  is  unambiguous 

An  explanation  of  these  tasks  follows. 

Document  Content  Structure 


The  content  structure  is  guided  by  either  SECNAVINST  3560.1  or 
MIL-STD-1679 .  Each  specifies  the  section  and  paragraph  titles  required 
for  each  document  and  provides  guidelines  for  the  content  of  each  para¬ 
graph.  This  review  ensures  that  each  paragraph  addresses  the  specified 
content.  The  reviewer  does  not  have  to  check  style  or  format;  he  checks 
for  technical  accuracy  only  to  the  extent  of  determining  that  the  specified 
requirements  are  addressed. 

Document  Boundary  Description 

This  review  determines  whether  the  boundaries  of  a  document  are  properly 
described  and  observed.  A  document  can  easily  extend  into  adjacent  areas 
rather  than  stopping  at  the  prescribed  limit.  While  this  extra  information 
is  nice  to  know,  the  document  should  reference  the  precedent  document  rather 
than  duplicating  the  information.  The  problem  with  duplicating  information 
is  that  it  increases  the  costs  for  maintaining  documents  over  the  life  cycle 
of  the  system  when  the  duplicated  material  is  changed. 

Document  Traceability 


This  review  ensures  that  al]  high-level  requirements  are  properly 
contained  in  lower-level  documents  and  that  all  requirements  in  low-level 
documents  are  derived  from  higher-level  requirements.  This  can  be  accom¬ 
plished  manually  or  with  the  support  of  a  computer  program  to  keep  an 
accounting  of  each  requirement,  such  as  the  functional  cross-verification 
reference  index  (FCVRI)  available  on  Share/7  at  FCDSSA,  San  Diego.  This 
review  requires  that  each  document  be  decomposed  into  element  titles  and 
element  descriptions,  and  that  these  be  traced  to  elements  in  both  higher 
and  lower  documents.  The  result  of  this  review  produces  document  cross- 
reference  reports.  Figures  3-1,  3-2,  and  3-3  provide  examples  of  an  SOS- 
to-SOD  downward  cross-reference,  an  SOD-to-SOS  upward  cross-reference,  and 
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Figure  3-3.  EXAMPLE  OF  A  PPS-TO-SOD  UPWARD  CROSS-REFERENCE 


f 


a  PPS-to-SOD  upward  cross-reference.  As  part  of  this  review,  the  V&V  agent 
shall  provide  the  following  reports: 

•  SOS-to-SOD  upward  and  downward  cross-ref erence 

•  SOD-to-PPS  upward  and  downward  cross-reference 

•  Summary  of  unsatisfied  requirements 

•  Summary  of  extraneous  elements 

In  addition,  horizonal  traceability  should  be  determined  between  the  PPS 
and  each  IDS  to  verify  that  each  interface  defined  in  the  IDSs  is  also 
contained  in  the  PPS. 

Document  Completeness 


This  review  ensures  that  all  requirements  and  specifications  from  the 
precedent  document  are  fully  described  in  the  detail  required.  This  facil¬ 
itates  the  preparation  of  the  traceability  reports  and  the  SECNAVINST  3560.1 
checkoff  lists.  When  all  elements  listed  in  SECNAVINST  3560.1  are  satis¬ 
fied  and  all  elements  from  the  precedent  documents  are  completely  described, 
the  document  will  be  complete. 

Document  Consistency 


This  review  provides  a  horizontal  review  of  the  document.  Internal 
consistency  means  even  and  equal  development  of  all  material.  This  review 
can  be  an  extension  of  the  completeness  review  and  should  use  the  trace- 
ability  report.  It  should  identify  any  requirement  that  is  being  traced 
only  to  the  introduction  of  a  document  and  is  not  amplified  in  the  body 
of  the  document.  It  seeks  to  ensure  that  equally  important  elements 
receive  equal  treatment. 

Document  Clarity 


This  review  is  conducted  to  ensure  that  the  document  is  clear.  Abbre¬ 
viations  and  acronyms  should  be  defined  when  first  used. 

3 . 1 . 1 . 4  Document  Review  Tasks 

The  following  actions  are  to  be  taken  upon  delivery  of  the  SOS,  SOD, 

PPS,  and  IDSs  to  the  FCDSSA  PAG  program  manager. 

•  The  program  manager  receives  the  document  and  logs  it  into  his 
document  library. 

•  The  program  manager  designates  responsible  reviewers  for  the  format 
and  content  review,  determines  the  schedule  for  the  review  and  whether 
the  comprehensive  review  will  be  conducted  in  parallel  or  in  series 
with  the  content  and  format  review,  and  distributes  the  document. 

•  The  designated  reviewers  conduct  the  format  and  content  reviews. 
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•  The  V&V  agent  conducts  the  comprehensive  review,  receives  all 
comments  from  the  format  and  content  review,  consolidates  all 
comments,  and  delivers  the  consolidated  comments  and  an  assessment 
of  overall  document  quality  to  the  program  manager. 

•  The  program  manager  reviews  and  assesses  the  comments  received  from 
the  V&V  agent  and  transmits  them  to  the  PAG  software  developer  with 
appropriate  instructions. 

•  The  PAG  software  developer  updates  the  document  and  redelivers  it 
as  necessary. 

•  The  program  manager  determines  the  extent  of  further  review  required 
He  may  start  another  complete  review  or  may  only  verify  the 
incorporation  of  comments  with  the  assistance  of  the  V&V  agent. 

•  When  the  program  manager  approves  the  document,  he  places  it  under 
informal  configuration  control  until  it  is  formally  approved  after 
the  PDR. 

3.1.2  Design  Reviews  and  Audits 


Before  any  phase  of  the  software-development  process  is  completed, 
either  a  design  review  or  an  audit  is  conducted  to  ensure  that  the  documents 
or  products  that  constitute  the  baseline  for  the  next  phase  are  satisfactory 
For  the  requirements-def inition  phase,  that  function  is  fulfilled  by  the 
PDR. 


The  PDR  is  a  formal  review  of  the  documents  that  constitute  the 
allocated-Part  X  baseline.  The  responsibility  for  establishing  and  schedul¬ 
ing  PDRs  lies  within  PME-109.  Several  PDRs  may  be  conducted  during  the 
requirements-def inition  phase  to  review  products  as  they  are  developed; 
this  is  done  at  the  conclusion  of  the  document  review  discussed  in  Section 
3.1.1. 

At  the  conclusion  of  the  requirements-def inition  phase,  the  SOS,  SOD, 
PPS,  and  IDSs  should  all  have  been  the  subject  of  at  least  one  PDR.  The 
result  of  these  reviews  should  be  a  clear  definition  and  establishment  of 
the  following: 

•  Detailed  performance  requirements 

•  Program  construction,  including  an  identification  of  subprogram, 
program  support  functions,  general  supervisory  functions,  program 
execution  and  operation,  and  types  of  stores  and  service  routines 

•  Input,  output,  and  processing  requirements  for  each  of  the  program 
functions  or  tasks 

•  Consoles,  console  modes,  and  number  of  consoles  on-line  for  differ¬ 
ent  conditions  of  systems  operation 

•  Functions  implemented  for  operator  support 

•  Interfaces  with  other  systems,  peripherals,  and  operators 

•  I/O  utilization  plans 
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•  Computer  resource  requirements,  i.e.,  computer  memory,  processing 
time,  and  input  and  output  allocations,  and  their  expected 
utilization 

•  Traceability  of  all  requirements  upward  and  downward  between  the 
SOS,  SOD,  and  PPS 

•  Traceability  of  requirements  between  the  PPS  and  the  IDSs 

The  FCDSSA  program  manager  for  PAG  software  development  has  the 
responsibility  to  establish  the  plan  for  conducting  each  PDR.  Advance 
planning  is  essential  for  a  meaningful  review.  The  following  paragraphs 
address  activities  that  are  to  be  performed. 

FCDSSA,  San  Diego,  shall  develop  the  PDR  plan.  They  shall  transmit  a 
copy  of  the  document  being  reviewed  to  each  reviewing  agency  via  official 
correspondence  approximately  40  days  in  advance.  The  correspondence  shall 
also  contain  the  following  information: 

•  Date  and  location  of  the  PDR 

•  Review  objectives 

•  Review  agencies  and  associated  review  responsibility 

•  Response  date  for  comments 

•  Standard  form  for  providing  comments  (if  desired) 

•  Name  and  address  of  individual  who  is  to  receive  the  comments 

Attendees  at  each  PDR  shall  include,  but  will  not  necessarily  be 
limited  to,  the  following  organizations: 

•  NAVELEX,  PME-109 

•  NAVSEA  612 

•  FCDSSA,  San  Diego 

•  V&V  agent 

•  Software  developer 

•  OPTEVFOR 

•  Naval  Ocean  System  Center  (NOSC) 

•  PAG  IDS  interfacing  agencies 

The  V&V  agent  shall  be  responsible  for  receiving,  logging,  and  con¬ 
solidating  all  comments,  and  providing  them  to  FDCSSA  and  the  software 
developer.  The  software  developer  shall  provide  written  responses  to  FDCSSA 
for  all  comments  received.  FCDSSA,  with  support  from  the  V&V  agent,  shall 
evaluate  the  responses  prior  to  the  PDR.  If  necessary,  the  responses  shall 
be  revised  by  the  software  developer  until  they  are  satisfactory. 

The  agenda  for  the  PDR  shall  be  established  by  FDCSSA.  The  meeting  should 
be  structured  to  derive  maximum  benefits  from  the  attendees  in  the  minimum 


amount  of  time.  It  is  recommended  that,  where  possible,  splinter  meetings 
be  held  on  specific  topics  to  reduce  the  length  of  the  PDR ,  with  reviewers 
attending  splinter  meetings  in  their  area  of  expertise.  A  suggested 
division  of  agenda  topics  is  as  follows: 

•  Hardware-related  agenda  items 

•  Software-related  agenda  items 

•  Management-related  agenda  items 

•  Operational  agenda  items 

The  PAG  software  developer  is  responsible  for  submitting  minutes  of 
the  PDR  to  FCDSSA  within  two  weeks  after  the  completion  of  the  PDR.  These 
minutes  shall  contain  the  following: 

•  All  comments  received  and  the  final  response 

•  All  action  items  arising  from  the  PDR,  including  responsible 
agencies,  individuals,  and  due  dates 

•  Significant  items  discussed  during  the  PDR 

The  V&V  agent  shall  monitor  the  incorporation  into  the  applicable 
document  of  all  changes  either  agreed  to  during  the  PDR  or  resulting  from 
PDR  action.  Upon  incorporation  of  all  changes,  NAVSEA  612  will  establish 
the  al located-Part  I  baseline. 

3.1.3  Configuration  Management 

Configuration  management  is  a  discipline  that  applies  technical  and 
administrative  direction  and  surveillance  (1)  to  identify  and  document  the 
functional  and  physical  characteristics  of  a  configuration  item,  (2)  to  con¬ 
trol  changes  to  those  characteristics,  and  (3)  to  record  and  report  change 
processing  and  implementation  status.  The  discipline  of  configuration 
management  is  made  up  of  the  following  elements: 

•  Configuration  identification  -  the  precise  identification  of 
approved,  or  conditionally  approved,  technical  documentation  for  a 
configuration  item  as  set  forth  in  specifications,  drawings,  and 
associated  lists 

•  Configuration  control  -  the  systematic  evaluation,  coordination, 
approval  or  disapproval,  and  implementation  of  all  approved  changes 
to  a  configuration  item  after  formal  establishment  of  its  con¬ 
figuration  baseline 

•  Configuration  status  accounting  -  the  recording  and  reporting  of 
information  needed  to  manage  a  configuration  effectively,  including 
a  listing  of  the  approved  configuration  identification,  the  status 
of  proposed  changes  to  the  configuration,  and  the  implementation 
status  of  approved  changes 


The  requirements-defimtion  phase  Begins  with  the  establishment  of  the 
functional  baseline  and  ends  with  the  establishment  of  the  al located-Part 
I  baseline.  Hence,  only  the  TOR  and  IDR  are  conf iguration-managed  at  the 
beginning  of  this  phase;  the  SOS,  SOD,  PPS,  and  IDSs  are  added  to  them 
during  this  phase.  When  these  documents  are  all  approved,  the  requirements- 
definition  phase  is  completed  and  the  program-design  phase  begins. 

CM  during  the  program-design  phase  is  confined  to  the  activity  associ¬ 
ated  with  the  CM  of  the  documents  noted  in  the  preceding  paragraph.  Each 
document  is  treated  as  a  separately  conf iguration-control led  item  that  can 
be  changed  only  by  approval  of  an  engineering  change  proposal  (ECP) .  The 
following  paragraphs  discuss  the  process  of  ECP  development  and  approval. 

An  ECP  may  be  originated  by  FCDSSA,  the  software  developer,  the  V&V 
agent,  or  any  other  agency  associated  with  PAG  software  development  who 
establishes  a  need  for  change.  Changes  should  be  limited  to  those  which 
are  necessary  or  offer  a  significant  benefit  to  the  Navy,  such  as  changes 
to  accomplish  the  following: 

•  Correct  deficiencies 

•  Satisfy  changes  in  operational  requirements 

•  Cause  life-cycle-cost  savings 

•  Improve  schedules 

The  originating  agency  shall  prepare  all  ECPs  on  DD  Form  1692  in 
accordance  with  DoD-STD-480A.  The  form  should  contain  the  following 
information : 

•  Originator's  name  and  organization 

•  Baseline  affected 

■  Title  of  change 

•  Priority 

•  Description  of  change 

•  Need  for  change 

•  Effect  on  cost  (if  known) 

•  Effect  on  schedule  (if  known) 

•  Effect  on  other  configuration  items  (if  known) 

The  last  three  items  may  not  be  .-...own  completely  or  at  all  to  the 
originator,  particularly  if  the  originator  is  not  the  software  developer. 
This  should  not  be  a  deterrent  to  submitting  ECPs,  however. 

The  originator  shall  also  submit  a  specification  change  notice  (SCN) 
for  each  configuration-managed  document  affected  by  the  proposed  ECP.  SCNs 
shall  be  prepared  on  DD  Form  1696  in  accordance  with  MIL-STD-480.  They 
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It  is  during  the  program-design  phase  *  hat  programming  standards  and 
conventions  are  written  by  the  software  developer.  These  standards  and 
conventions  will  be  used  by  the  software  developer  during  the  actual  coding 
of  the  computer  program.  Typical  programming  standards  and  conventions 
could  include  standard  commenting  procedures,  naming  conventions,  limits 
on  code  complexity,  and  limitations  on  branching  statements. 

Test  and  evaluation  activity  during  the  program-design  phase  com¬ 
p-rises  the  specification ,  development,  and  certification  of  test  tools 
required  in  subsequent  ; bases  and  the  ievelopment  of  test  documentation 
for  subprogram-module  test  (SP/MT)  and  function  test  (FT).  Test  tools 
should  be  certified  ir.  the  phase  prior  to  their  actual  use.  For  example, 
a  test  tool  required  for  the  j -roqram-code -and-debug  phase  should  be  certi¬ 
fied  during  th*.  program-design  phase,  and  a  test  tool  required  for  the 
program-acceptunco-test  phase  should  be  certified  during  the  program-code- 
and-debug  phase.  Similarly,  test  documentation  is  developed  in  the  phase 
before  the  one  ir.  which  it  is  required. 


3.2.1  Documentation  Review 

The  PLS  ar.  1  the  DBuD  will  both  be  reviewed  as  described  in  Section 
3.1.1,  i.o.  ,  they  will  he  subioct  to  .»  format,  content,  and  comprehensive 
review.  The  governing  requirements  for  their  review  are  as  follows: 

•  PDS  -  MI I. -STD- 1670  and  DID  Dl-F-2138 

•  DBDD  -  MIL-STD- li .79  and  DID  DI-S-2140 

As  with  the  PPS  and  IDSs,  the  content  review  requires  that  a  document 
traceability  analysis  be  performed.  This  requires  that  each  element  in  the 
PDS  be  identified,  tided,  and  traced  ruck  to  a  requirement  in  the  PPS. 
Similarly,  every  element  previously  idcnt. i  f  i < -d  in  the  PPS  must  be  traceable 
to  an  element  in  the  PDS.  This  assures  that  ill  design  features  included 
are  due  to  performance  requi  i  •  motifs  and  th.it  ill  performance  requirements 
are  reflected  in  design  clem*  nts.  Th-  Vt,V  :.t  shall  submit  a  report  to 
FCDSSA  contain  in- 1  the  f.,  1  lowing  as  -i  result  :  the  traceability  analysis: 

•  A  reqert.  of  IT'S-t.o-i'DS  dowtiwat  :  t  l  urea:  llltv 

•  A  to}  art  of  PDS-to-M'S  upward  1.  .•  a  aul  ■  i  1  l  tv 

•  A  summary  of  ur.satisfie  1  iesiun  requirements 

•  A  summary  of  extt  uneous  elements 

When  i  i:viewing  th-  PDS,  the  V-iV  agent,  shall  verify  that  the  software 
dove  1  or  ha.-  designed  the  com}. uter  program  by  a  systematic  top-down  method. 
To  i ccomj  lish  ‘his,  the  VSV  agent  shall  verify  that  the  following  features 
are  ref )  •••'fed  in  th--  PDS: 

•  Th"  ie.jigri  is  a  hierarchical  structure  of  identifiable  programs , 
su;.:  t  ogi  ams  ,  modules,  procedures  ,  and  routines. 

•  i'ho  hiii.-st  level  of  ■  ontrnl  lies  at  the  top  of  the  hierarchy. 
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•  The  computational  or  arithmetic  functions  reside  at  the  lower 
levels . 

•  The  program  is  divided  into  constituent  parts  and  then  these  consti¬ 
tuent  parts  are  broken  down  into  their  constituents. 

•  Each  level  of  design  is  continued  until  a  level  is  reached  where 
there  is  no  subordinate  level. 

•  A  lower  level  does  not  call  a  higher  level. 

The  V&V  agent  shall  review  the  proposed  design  to  ascertain  the  capa¬ 
bility  to  support  the  computational  load  imposed  by  maximum  operation  of 
all  functions  required  to  be  simultaneously  serviced.  If  modeling  is 
required  in  the  performance  of  this  function,  the  V&V  agent  should  first 
obtain  approval  from  FCDSSA.  The  V&V  agent  shall  also  verify  that  the 
resource  allocations  for  computer  memory,  processing  time,  and  input  and 
output  channels  contained  in  the  PDS  are  within  the  resource  requirements 
in  the  PPS. 

During  this  phase  the  software  developer  shall  also  prepare  programming 
standards  and  conventions,  which  shall  be  reviewed  for  adherence  to  the 
programming  standards  and  programming  conventions  sections  of  MIL-STD- 
1679  and  for  conformity  to  other  established  standards  used  in  the  JTIDS 
program.  The  V&V  agent  should  specifically  verify  that  the  programming 
standards  prepared  by  the  software  developer  meet  the  requirements  of  the 
programming  standards  section  of  MIL-STD-1679  for  the  following  areas: 

•  Control  structures 

•  Included  copies  or  segments 

•  Entry-exit  structure 

•  Program  traceability 

•  Self-modification 

•  Recursive  programs 

•  Size 

•  Branching 

•  Re locatabil ity 

•  Indentation 

Similarly,  the  V&V  agent  shall  verify  that  the  programming  conventions 
meet  the  requirements  of  the  |  roqramminq  conventions  section  of  MII.-STD- 
.1679  for  the  following  areas: 

•  Naming 

•  Symbolic  constants  and  variables 

•  Mixed-mode  expressions 

•  Grouping 


•  Significant  digits 


•  Narrative  descriptions 

•  Source-record  formats 


3.2.2  Design  Reviews  and  Audits 


The  critical  design  review  (CDR)  is  held  at  the  conclusion  of  the 
program-design  phase  to  review  the  products  that  will  constitute  the 
allocated-Part  II  baseline.  The  purpose  of  the  CDR  is  to  ensure  that  the 
software  design  satisfies  the  performance  requirements  established  in  the 
PPS  an a  *_o  evaluate  the  FT  plans,  specifications,  and  procedures  prior  to 
conduct  of  the  FT  program.  Additiona lly ,  the  CDR  serves  to  focus  management 
and  designer's  attention  on  the  products  of  the  design  effort  and  quality 
of  th-  design  prior  to  initiating  programming  activity.  The  CDR  is  there¬ 
fore  scheduled  by  FCDSSA  at  a  point  in  the  development  cycle  coincident 
vit.n  completion  of  design  documents.  The  basic  documents  subject  to  the 
CDR  process  consist  primarily  of  the  PDS  supported  by  the  DBDD  and  FT 
plans,  sj  -.reifications,  and  procedures. 

At  the  option  of  FCDSSA,  the  CDR  may  be  conducted  in  several  discrete 
parts,  each  part  oriented  toward  evaluation  and  acceptance  of  the  PDS  for 
operational  programs,  maintenance  and  diagnostic  programs,  and  training 
and  simulation  programs  or  computers.  The  specific  elements  of  the  CDR 
process  shall  bn  established  by  FCDSSA  as  part  of  the  initial  program¬ 
planning  process.  As  each  CDF  is  completed  the  documents  reviewed  during 
the  CDR  will  be  baselined  and  changes  to  the  documents  formally  controlled. 
When  the  last  CDR  has  been  completed,  the  allocated-Part  II  baseline  will 
be  established. 

The  primary  product  of  the  CDR  is  the  formal  identification  of  computer 
program  documentation  that  wi i 1  be  released  for  coding  and  testing.  The 
CDF  will  be  oriented  toward  the  following: 

•  Establishing  program-design  compatibility  with  performance 
spot :  i f icutions . 

•  Establishing  system  compatibility  by  review  of  all  interfaces 
between  computer  proqrams  and  between  subprograms  by  analysis  of 
detailed  flow  diagrams  and  other  descriptive  documentation 

•  Review  arid  evaluation  of  data  base  design 

•  Review  of  design  integrity  by  review  or  available  analytical  data 
in  the  form  o.‘  flow  diagrams,  logic  diagrams,  algorithms,  and 
storage  al locations 

•  Review  ...nd  •  valuation  of  the  function  test  plan,  test  specifications, 
and  test,  procedures 

FCDSPA,  Sal.  Diego,  shall  develop  the  CDR  plan.  They  shall  transmit  a 
••  -i  ;  ■>:  the  L’DS  at.d  DBDD  to  e.n  h  reviewing  agency  via  official  correspondence 
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approximately  40  days  in  advance.  The  correspondence  shall  also  contain 
the  following  information: 

•  Date  and  location  of  the  CDR 

•  Review  objectives 

•  Review  agencies  and  associated  review  responsibility 

•  Response  date  for  comments 

•  Standard  form  for  providing  comments  (if  desired) 

•  Name  and  address  of  individual  who  is  to  receive  the  comments 

Attendees  for  each  CDR  shall  include,  but  will  not  necessarily  be 
limited  to,  the  following  organizations: 

•  NAVELEX,  PME-109 

•  NAVSEA  612 

•  FCDSSA,  San  Diego 

•  V&V  agent 

•  Software  developer 

•  OPTEVFOR 

•  NOSC 

•  PAG  IDS  interfacing  agencies 

The  responsibilities  of  the  software  developer,  the  V&V  agent,  and 
FCDSSA  during  the  planning,  conduct,  and  reporting  of  the  CDR  are  identical 
to  the  responsibilities  for  these  organizations  described  in  Section  3.1.2 
with  respect  to  the  PDR.  Upon  satisfactory  completion  of  the  CDR,  NAVSEA 
612  will  establish  the  allocated  baseline. 

3.2.3  Configuration  Management 

CM  during  the  program-desian  phase  is  similar  to  the  CM  conducted 
during  the  requirements-def inition  phase  described  in  Section  3.1.3.  The 
only  change  is  the  increased  number  of  documents  under  the  discipline  of 
CM.  The  additional  documents  are  as  follows: 

•  SOS 

•  SOD 

•  PPS 

•  IDSs 

The  procedures  and  respons ibi 1 l t l e:;  for  processing  ECPs  to  documents  being 
con  figuration -managed  is  the  same  as  that  described  in  Section  3.1.3. 


3.2.4  Test  and  Evaluation 


As  shown  in  Figure  2-2,  the  test  and  evaluation  activity  during  the 
program-design  phase  is  as  follows: 

•  Test  tool  specification,  development,  and  certification  for  SP/MT 
and  FT 

•  Test  documentation  development  for  SP/MT  and  FT 

As  mentioned  previously,  the  general  policy  is  that  all  items  required 
for  a  particular  phase  of  testing  are  certified  or  approved  in  the  previous 
phase.  Since  the  following  phase  includes  the  conduct  of  SP/MT  and  FT,  all 
test  tools,  including  simulation  programs,  needed  to  support  these  tests 
must  be  certified  in  the  program-design  phase,  and  test  documentation  for 
SP/MT  and  FT  must  be  developed  and  approved  during  this  phase. 

SP/MT  is  the  first  step  in  testing  a  computer  system.  The  purpose  of 
this  test  is  to  demonstrate  that  the  internal  logic  cf  the  module  is  cor¬ 
rect.  This  is  accomplished  by  exercising  each  module  sufficiently  to 
verify  that  it  conforms  to  the  program  design.  Responsibilities  for 
developing  test  documentation  and  test  tools  for  SP/MT  are  listed  in  Table 
3-1.  There  is  a  minimum  of  formal  accountabi 1 ity  for  SP/MT  test  documenta¬ 
tion,  since  this  level  of  testing  is  normally  accomplished  by  the  individual 
programmers  with  the  software  developer's  organization  and  the  Government 
does  not  take  formal  delivery  of  the  software  until  after  the  completion 
of  the  FTs .  However,  ill  test  documentation  developed  by  the  software 
developer  should  be  available  -  tin.-  VaV  agent,  for  review. 
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Test  tools  for  SP/MT  are  developed  by  either  FCDSSA  or  the  software 
developer.  Normally,  existing  test  tools  under  FCDSSA  cognizance  that 
require  modification  to  be  used  for  PAG  software  testing  will  be  developed 
by  FCDSSA.  New  test  to-'ls  are  usually  developed  by  the  software  developer. 
In  either  case,  all  test  tools  used  for  SP/MT  are  to  be  reviewed  and  veri¬ 
fied  by  the  V&V  agent  and  approved  by  FCDSSA  prior  to  the  completion  of  the 
program-design  phase. 

FTs  are  the  second  step  in  testing  the  PAG  software.  FTs  are  conducted 
to  validate  that  the  modules,  when  combined  into  operating  functions,  will 
perform  as  specified  in  the  PPS  and  PDS.  FTs  are  therefore  the  beginning 
of  testing  the  integrated  software  modules  within  the  PAG  computer.  An  FT 
is  conducted  for  each  function  allocated  to  the  PAG  in  accordance  with 
test  plans,  specifications,  and  procedures  prepared  by  the  software  develope 
and  approved  by  FCDSSA.  Table  3-2  lists  the  responsibilities  for  developing 
FT  test  documentation  and  test  tools.  This  level  of  testing  requires  that 
the  modules  be  operationally  combined  to  verify  the  overall  program  sub¬ 
system  operation  as  described  in  the  PPS  and  PDS.  Functions  are  executed 
and  tested  one  at  a  time.  There  is  no  simultaneous  execution  of  functions, 
so  FTs  are  the  simplest  level  of  integration-oriented  testing. 


Table  3-2. 

RESPONSIBILITIES  FOR  FT  TEST 

DOCUMENTATION  AND  TEST  TOOLS 

Item 

Responsibility 

Requirements 

Analysis 

Development 

Testing 

Reviewing 

Approval 

Test  plan 

FCDSSA 

Software 

N/A 

FCDSSA 

developer 

Pil 

Test  specif ication 

FCDSSA 

Sof  tware 

N/A 

V&V 

FCDSSA 

deve  1 0[  er 

agent 

Test  procedures 

Sot tware 

Software 

N/A 

V&V 

FCDSSA 

developer 

developer 

agent 

Test  tools 

FCDSSA/ 

FCDSSA/ 

FCDSSA/ 

V&V 

FCDSSA* 

software 

software 

software 

agent 

developer 

developer 

deve loper 

‘Certification. 

FT  test  plans,  specifications,  and  procedures  are  to  be  reviewed  by 
the  V&V  agent  for  conformity  to  MU, -STD-1679  and  the  following  applicable 
DIDs: 

•  Test  plan  -  DI-T-2142 

•  Test  specification  -  DI-E-2143 

•  Test  procedures  -  DI-T-2144 


The  FT  test  plan  will  define  the  total  scope  of  the  testing  to  be  per¬ 
formed,  containing  precise  statements  of  the  purpose,  scope,  and  schedule 
for  the  individual  test  being  planned.  In  reviewing  the  FT  test  plan,  the 
V&V  agent  should  specifically  verify  that  tests  are  planned  to  accomplish 
the  following: 

•  Ensure  error-free  linkage  of  each  module 

•  Ensure  that  each  tested  function  fully  satisfies  the  detailed 
performance  and  design  requirements 

•  Exercise  each  function  in  terms  of  input-output  performance  and 
ensure  that  the  results  satisfy  the  applicable  detailed  performance 
and  review  requirements 

•  Ensure  that  each  function  man-machine  interface  is  as  specified  in 
the  PPS 

•  Ensure  the  capability  of  the  subprogram  to  handle  erroneous  inputs 
properly  and  survive  them. 

In  addition,  a  traceability  analysis  shall  be  performed  by  the  V&V  agent  to 
ensure  that  each  requirement  in  the  PPS  and  PDS  is  covered  by  a  test  in 
the  FT  test  plan.  An  analysis  is  also  to  be  conducted  upward  from  the  FT 
test  plan  to  ensure  that  each  test  being  planned  is  traceable  to  either  a 
design  requirement  or  a  performance  requirement.  Results  of  each  analysis 
are  to  be  reported  to  FCDSSA. 

The  FT  test  specification  contains  test  specifications  for  each  test 
contained  in  the  FT  test  plan.  The  V&V  agent  shall  review  this  document 
to  ensure  the  following: 

•  Each  test  in  the  FT  test  plan  has  a  corresponding  test  specification 

•  Test  criteria  are  identified. 

•  Test  methods  are  explained. 

•  The  purpose  of  each  test  is  identified. 

•  The  software  to  be  tested  and  the  scope  of  each  test  are  identified. 

•  Support  requirements,  inputs,  required  accuracies,  expected  output, 
and  data  collection  methods  for  each  test  are  identified. 

The  FT  test  procedures  present  detailed  instructions  for  test  execu¬ 
tion  and  for  evaluation  of  the  results  of  each  level  of  testing  specified. 
They  are  developed  from  the  FT  test  specification  and  the  relevant  design 
document,  and  give  detailed  instructions  for  test  setup),  execution,  and 
evaluation  of  the  test  results.  The  V&V  agent  should  review  the  FT  test 
procedures  to  ensure  the  following: 

•  The  organization  or  structure  of  the  procedure  and  any  assumptions 
or  any  constraints  on  its  use  are  identified. 

•  Detailed  instructions  for  the  setup  and  operation  of  each  test  are 
presented . 


•  The  total  equipment,  manpower,  digital  processor  programs,  and 
supporting  documentation  required  for  each  operation  are  described. 

•  The  requirements  for  various  modes  of  operation  are  specified 
if  they  differ  from  the  total  requirements. 

•  Equipment  required  for  operation  is  identified  by  official 
nomenclature . 

•  Revisions  or  modif ications  to  required  equipments  are  specified, 
as  well  as  any  pretest  checkout  of  the  hardware  required. 

•  Materials  and  personnel  required  for  the  performance  of  the  test 
are  identified. 

•  The  test  setup  and  energizing  procedures  are  given. 

•  The  program  loading  procedure  is  given. 

Test  tools  for  FT  are  developed  by  either  FCDSSA  or  the  software 
developer,  as  with  SP/MT.  All  test  tools  used  for  FT  are  to  be  reviewed 
and  verified  by  the  VSV  agent  and  certified  by  FCDSSA  prior  to  the  comple¬ 
tion  of  the  program-design  phase.  Once  the  FT  test  tools  have  been 
certified,  they  shall  be  configuration-managed  by  FCDSSA  in  a  manner  similar 
to  that  for  the  PAG  documentation  and  computer  programs. 


3.2.5  Status  Reports 


During  the  program-design  phase  the  V&V  agent  shall  submit  monthly 
status  reports  to  FCDSSA  and  NAVSEA,  in  addition  to  those  submitted  as  a 
result  of  specific  activities,  such  as  document  or  ECP  review.  These 
reports  shall  contain  at  least  the  following: 

•  An  assessment  of  program  progress  with  respect  to  planned 
schedules 

•  A  description  of  outstanding  technical  problems  requiring  resolution 

•  A  summary  of  the  tasks  performed  during  the  past  month 

•  A  summary  of  the  tasks  planned  for  the  coming  month 


3.3  PROGRAM- CODE -AND- DEBUG  PHASE 

The  program-code-and-debug  phase  begins  with  the  establishment  of 
the  al located-Part  II  baseline  and  ends  with  the  establishment  of  the  pre¬ 
liminary  product  baseline.  Figure  3-2  shows  the  activity  that  occurs 
during  the  code-and-debug  phase.  The  object  of  this  phase  is  to  code  and 
debug  the  PAG  computer  program  in  accordance  with  the  PDS  and  DBDD,  and 
begin  the  initial  testing  of  the  program.  The  primary  computer  program 
documentation  produced  'luring  this  phase  is  as  follows: 

•  Program  design  document  (FDD) 

•  Program  package  document 
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The  PDD  contains  a  complete  technical  description  of  all  PAG  software 
subprogram  functions,  structures,  operation  environments,  operating  con¬ 
straints,  data  base  organization,  source  and  object  code  listing,  and 
diagrammatic  and  narrative  flows.  The  PDD  is  specifically  oriented  to 
programming  logic  and  programmer's  language.  Program  listings  are  referenced 
as  appendixes  to  the  PDD. 

The  program  package  document  consists  of  the  program-source  listing, 
an  error-free  source/object  listing  produced  by  an  assembly  or  compilation 
of  the  source,  a  complete  cross-reference  listing  produced  by  a  compila¬ 
tion  of  the  source,  and  any  data  necessary  to  cause  programs  to  run  properly, 
e.g.,  adaptation  data  or  program  parameter  values. 

Test  and  evaluation  activity  that  occurs  during  the  program-code-and- 
debug  phase  is  the  conduct  of  SP/MT  and  FT,  development  and  certification 
of  test  tools  for  program  acceptance  tests  (PATs),  and  development  of  test 
documentation  for  PATs. 

3.3.1  Documentation  Review 

The  PDD  will  be  reviewed  as  described  in  Section  3.1.1,  i.e.,  it  will 
be  subjected  to  a  format  review,  a  content  review,  and  a  comprehensive 
review.  The  governing  requirements  for  these  reviews  are  contained  in  MIL- 
STD-1679  and  DID  DI-S-2139. 

The  comprehensive  review  will,  include  a  traceability  analysis  between 
the  PDS  and  the  PDD.  This  requires  that  each  element  in  the  PDS  be  identi¬ 
fied,  titled,  and  traced  downward  to  an  element  in  the  PDD.  Similarly,  every 
element  in  the  PDD  must  be  traceable  to  an  element  in  the  PDS.  This  trace- 
ability  analysis  assures  that  all  design  elements  in  the  PDS  are  accounted 
for  in  the  PDD  and  that  all  procedures  ar.d  routines  in  the  PDD  can  be  traced 
to  a  design  element  in  the  FDS .  The  V&V  agent  shall  submit  a  report  to 
FCDS3A  containing  the  following: 

•  A  I  DS-t  '-PDD  .downward  cross-reference 

•  A  PDD-to-PDS  ur w  ird  cross-reference 

•  A  summary  of  unsatisfied  design  elements 

•  A  summary  ■  > f  extraneous  elements 

In  reviewing  the  PDD,  the  VS.V  agent  shall  verify  that  the  software 
developer  has  i  mt(  lemon  ted  the  ienign  in  the  PDS  in  a  systematic  top-down 
method.  •*!  1  tena  for  verifying  this  feature  are  contained  in  Section 
i .  .  1  atid  sh<  uld  be  used  l r  he  reviewing  ot  the  PDS. 

1 '  .  imjreti  ,il  tor  t  i.e  VfcV  agent  to  review  the  actual  code  produced 

by  ?  he  :'w,ii  ■  j.  ve]o|  et  at  the  completion  of  the  program-code-and-debug 
;  ;;a..e ,  o  t  s»  i*  >  t  amount  ()f  ,.0de  precludes  this  review.  However,  con- 

.  f-r  abb  t  tir.e  at.  b>  •  saved  by  detecting  errors  as  early  in  the  design  cycle 
m  :  •  •.  ;il  !•  .  T>  ac  -om]  1  i.,h  this  two  forms  of  manual  checking  shall  be 
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performed  by  the  software  developer  as  part  of  the  program-code-and-debug 
phase:  desk-checking  and  code  walkthroughs. 

DesK-checking  shall  be  performed  by  the  programmers  in  the  software 
development  organization.  It  can  include  a  number  of  check-out  efforts 
performed  manually,  but  it  normally  includes  one  or  all  of  the  following: 

•  Reviewing  a  source  list  for  faults 

•  Performing  arithmetic  calculations  manually  to  verify  program  out¬ 
put  values 

•  Manually  simulating  program  execution  to  verify  program  logic  and 
data  flow. 

Since  these  activities  are  performed  at  the  individual  programmer's  desk, 
no  V&V  agent  review  is  appropriate,  but  management  policies  of  the  software 
developer  should  require  that  desk-checking  be  performed. 

Code  walkthroughs  are  a  peer  group  review  of  the  code  as  it  is  developed. 
They  are  to  be  conducted  by  the  software  developer  as  the  code  is  being 
developed.  The  following  guidelines  are  to  be  followed  in  the  conduct  of 
code  walkthroughs : 

•  The  walkthrough  is  to  be  an  informal  review  of  the  code  conducted 
by  the  peers  of  the  individual  programmer. 

•  The  size  of  the  code  under  review  should  not  exceed  the  amount  that 
can  be  reviewed  in  approximately  one  hour. 

•  The  walkthrough  is  to  be  held  after  the  code  has  been  compiled. 

The  purpose  of  the  code  walkthrough  is  to  conduct  a  review  of  the  code  in 
a  nondefensive  environment  for  the  programmer.  Software  managers  as  such 
are  normally  discouraged  from  attending  code  walkthroughs  to  avoid  the 
appearance  of  a  formal  performance  review.  Attendance  of  the  V&V  agent  at 
code  walkthroughs  is  likewise  not  conducive  to  achieving  the  proper  setting. 

The  responsibility  of  the  V&V  agent  with  respect  to  code  walkthroughs 
is  only  to  ensure  that  the  software  developer  is  conducting  code  walk¬ 
throughs.  In  addition,  the  software  developer  shall  furnish  source  listings 
to  the  V&V  agent  after  code  has  been  reviewed  at  a  code  walkthrough  and 
found  to  be  acceptable.  The  V&V  agent  shall  review  these  listings  to 
ensure  the  following: 

•  Approved  programming  standards  and  conventions  are  being  followed. 

•  The  code  is  completely  written  in  CMS-2Y  (Subset  0) .  Use  of  direct 
code  or  assembly  code  is  prohibited  without  the  written  permission 
of  PME-ino. 

•  A  top-down  structured  upp roach  is  being  followed. 

•  The  code  is  not  too  complex  for  the  intended  application. 


•  Comments  are  included  throughout  the  code  so  that  program  mainte¬ 
nance  is  improved. 

•  The  code  is  free  of  logical  faults  and  design  faults. 

•  Computer  resource  allocations  for  computer  memory,  processing  time, 
and  input-output  channels  are  being  observed  during  the  program- 
code-and-debug  phase. 

All  review  comments  shall  be  provided  to  the  software  developer  for  resolu¬ 
tion.  Problems  that  cannot  be  resolved  directly  between  the  software 
developer  and  the  v&V  agent  shall  be  resolved  by  FCDSSA. 

3.3.2  Design  Reviews  and  Audits 

The  preliminary  product  baseline  configuration  audit  (PPBCA)  is  held 
at  the  completion  of  the  program-code-and-debug  phase.  The  purpose  of  the 
audit  is  to  provide  formal  evaluations  of  the  tested  program  to  verify 
achievement  of  the  requirements  in  the  PPS  and  PDS,  ensure  that  the  product 
under  test,  the  computer  program,  is  consistent  with  the  program  documenta¬ 
tion,  and  verify  that  all  modifications  found  necessary  as  a  result  of  FT 
have  been  properly  incorporated  into  the  program  package  and  supporting 
documentation . 

FCDSSA  has  the  responsibility  to  schedule  and  conduct  the  PPBCA. 

It  shall  notify  attendees  by  official  correspondence  at  least  30  days  in 
advance  of  the  scheduled  date.  Attendees  at  the  PPBCA  shall  include  the 
fol lowing : 

•  NAVELEX,  P ME- 109 

•  NAVSEA  612 

•  FCDSSA,  San  Diego 

•  V&V  agent 

•  Software  developer 

•  OPTEVFOR 

•  NOSC 

•  PAG  IDS  interfacing  agencies 

Unlike  the  design  review,  the  PPBCA  requires  attendees  to  review  the  PAG  com¬ 
puter  program  at  the  lowest  level.  Hence,  attendees  should  be  limited  to  those 
individuals  with  the  expertise  to  conduct  a  thorough  review  at  this  level. 
FCDSSA  shall  ensure  that  the  software  developer  has  key  programming  and 
design  personnel  in  attendance  or  on  call  to  provide  assistance  as  necessary. 
The  V&V  agent  has  the  responsibility  to  be  the  technical  lead  for  the 
PPBCA  and  to  describe  all  findings  of  the  PPBCA  audit  team  for  transmittal 
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by  FCDSSA  to  the  software  developer.  The  V&V  agent  shall  ensure  that 
following  documents  are  reviewed  in  accordance  with  the  following  criteria: 


i 


•  FT  test  reports  document  the  results  of  function  testing  and  sum¬ 
marize  the  discrepancies  between  the  intended  design  and  the 
actual  program  capability.  Review  of  FT  reports  as  a  part  of  the 
PPBCA  should  ensure  the  following: 

• •  The  tests  were  conducted  in  accordance  with  the  approved  test 
procedures . 

••  All  deviations  from  the  FT  specification  and  function  test 
procedures  are  clearly  described,  and  rationale  provided  of 
the  necessity  for  the  deviation. 

••  Each  test  discrepancy  is  adequately  described  and  is  supported 
by  statements  of  its  significance  and  its  impact  on  the  program. 

••  The  test  report  includes  an  overall  analysis  of  the  functional 
performance  of  the  program  tested. 

••  The  number  of  software  errors  is  within  the  limits  specified  by 
MIL-STD-1679. 

••  The  number  of  object  patches  is  within  the  limits  specified  by 
MIL-STD-1679. 

••  Test  data  are  maintained  in  a  test  data  file  for  future  reference. 

•  The  PDD  provides  a  complete  technical  description  of  all  digital 
processor  subprogram  functions,  structures,  operation  environments, 
operating  constraints,  data  base  organization,  source  and  object 
code  listing,  and  diagramatic  and  narrative  flows.  Development  of 
the  PDD  is  based  on  the  requirements  of  the  PDS  and  the  common 
data  items  contained  in  the  DBDD.  During  conduct  of  the  PPBCA 

the  review  and  evaluation  of  the  PDD  should  ensure  the  following: 

•*  All  of  the  major  functions  (subprograms)  described  in  the 
program  design  specification  are  presented  in  the  PDD. 

••  All  program  logic  is  fully  described. 

••  The  detailed  design  of  each  subprogram  is  fully  described. 

••  For  each  major  procedure  or  subroutine  a  flow  diagram  is  pro¬ 
vided  that  specifies  all  operations  performed  and  includes  all 
equations  used  in  mathematical  computations. 

••  The  PDD  contains  the  required  level  of  detail  for  all  sub¬ 
program  tables,  variables,  flags,  and  indexes. 

••  The  PDD  contains  a  complete  listing  of  all  local  and  common 
data  base  references  and  the  location  of  each  reference. 

•*  A  brief  description  and  graphic  representation  of  each  input 
and  output  message,  card  format,  and  tape  format  is  presented. 

••  All  system  library  subroutines  are  listed. 

••  System  conditions  that  must  exist  for  subprogram  initiation  are 
described . 
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All  known  or  anticipated  subprogram  limitations  are  summarized. 


••  An  interface  description  is  provided  showing  the  *.  motional 

relationships  of  the  subprogram  with  other  subprograms,  system 
subroutines,  and  executives  with  which  it  interfaces. 

••  The  computer  resource  limitations  established  in  the  PPS  and 
PDS  are  being  met. 


•  The  program  package  document  includes  the 
digital  processor  program  (card  decx  ,  mag 
of  the  program  material  items  such  as  the 
object  listings,  and  other  data  necessary 
running.  When  reviewing  this  data  during 
should  ensure  the  following: 


source  form  of  the 
tape,  paper  tape)  and  all 
sources  listing,  source/ 
for  proper  program 
PPBCA,  the  audit  team 


••  All  items  making  up  the  ;  rogram  package  are  properly  identified. 

••  The  source  form  of  the  digital  processor  program  is  compatible 
with  the  equipment  in  the  user's  facility. 

•*  The  source  program  listing  is  an  exact  duplication  of  the  data 
contained  cn  the  magnetic  tape  or  card  deck  that  includes  the 
source  form. 

••  The  source/object  listing  is  error-free  and  reflects  an 
exact  presentation  of  the  source  and  object  programs  (the 
source/object  listing  is  provided  by  the  supporting  compiler 
or  assembler) . 

••  The  proqram  package  contains  a  cross-referenced  table  of  state¬ 
ments  that  make  up  the  digital  processor  program.  Statements 
are  cross-referenced  by  mnemonic  labels  and  the  address  of  each 
reference  to  the  label. 

••  Object  patches  within  the  limits  of  MlL.-STD-lf.79  are  properly 
documented  as  part  of  the  configuration  item. 

••  PAT  test  documentation  -  The  PAT  demonstrates  the  total  oper¬ 
ating  capability  of  th"  PA.  with  all  its  functions  integrated 
into  one  complete  program.  The  criteria  to  be  considered  in  a 
review  of  the  PAT  test  documentation  a  re  the  same  as  those 
contained  in  Section  1.1.4. 


Minutes  of  the  PPBCA  containing  .ill  findings  of  th<  audits  shall  be 
submitted  to  t’CDSSA  within  tv.  weeks  b  the  VfcV  .  The  V&.V  agent  shall 

monitor  the  coin;  lotion  of  all  i  ■  .  t.-miit  i  tion  items  cssary  to  complete 
the  audit  satisfactorily.  When  t  hi  ■  is  i  ccom{  lisle.),  KPCS.SA  will  request 
NAVSEA  61 2  to  establish  the  preliminary  j  lodu.t  baseline. 

3.3.3  Con f igura t i on  Management 

CM  during  the  program— design  ;  h.ise  is  concerned  with  tv  *  types  of 
products :  documentation  i:.d  com;  uter  •  n .  ir.iim. .  CM  :  iecument  at  i-»n  is 


similar  to  t  hat  described  m  Section  3.1.3,  except  that  the  following 
additional  documents  are  under  CM: 


ri 


•  i'DS 

•  DBDD 

The  procedures  and  responsibilities  for  piocessing  ECPs  to  documents  being 
con f igura t ion-managed  are  the  same  as  these  described  in  Section  3.1.3. 

Computer  programs,  like  documentation,  require  the  discipline  of  CM 
once  they  have  been  approved  by  the  customer  and  established  as  a  baseline. 
This  approval  occurs  at  the  establishment  of  the  preliminary  product  base¬ 
line.  During  the  program-code-and-debug  phase,  the  computer  program  has 
not  yet  been  defined  as  a  baseline,  and  formal  code  control  procedures 
mandated  by  the  Government  are  not  necessary.  However,  it  is  incumbent  upon 
the  software  developer  to  use  proper  CM  procedures  internally.  Thus,  it 
is  required  that  the  software  developer,  at  the  completion  of  SP/MT,  adopt 
procedures  for  code  control  internal  to  his  organization.  These  procedures 
shall  require  that  the  programmer  be  prohibited  from  modifying,  either  by 
patch  or  recompile,  any  code  that  has  successfully  passed  SP/MT  unless  a 
written  software  problem  report,  or  an  ECP,  exists  to  justify  the  change. 

The  format  for  the  software  problem  reports  may  be  of  the  developer's  own 
choosing.  The  procedures  shall  be  furnished  to  the  V&V  agent,  who  shall 
be  responsible  for  auditing  their  use. 

/ 

3.3.4  Test  and  Evaluat ion 

As  shown  in  Figure  2-2,  the  test  and  evaluation  activity  during  the 
program-code-and-debug  phase  is  as  follows: 

•  Conduct  of  SP/MT  and  FT 

•  Development  of  PAT  test  documentation 

•  Test  tool  specification,  development ,  and  certification  for  PAT 

3.  3.4.1  Conduct  of  SP/MT 


Both.  SP  'MT  and  FT  are  conducted  during  this  phase.  SP/MT  is  accom¬ 
plished  primarily  by  individual  programmers  using  module  drivers,  such  as 
the  static  environment  test  tool  (SETT)  on  Share/7  at  i'CDSSA.  The  primary 
purpose  of  SP/MT  test:.-  is  to  demonstrate  that  the  internal  logic  of  the 
module  is  correct.  Because  of  the  nature  of  the  SP/MT  tests,  the  witness¬ 
ing  of  each  SP/MT  test  by  the  V&V  agent  is  impractical.  The  software  devel¬ 
oper,  however,  shall  provide  a  duplicated  Share/7  file  of  the  code  under¬ 
going  SP/MT  to  the  V&V  agent,  so  that  he  can  independently  test  the  code 
with  the  SETT.  The  V&V'  agent  shall  not  attempt  to  duplicate  test  cases 
used  by  the  software  developer,  but  shall  concentrate  on  the  following: 

•  Testing  for  unexpected  inputs 
’  Testing  for  boundary  conditions 


I 

y 
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•  Testing  unusual  test  cases 

•  Testing  accuracy  of  all  equations  over  the  entire  range  of  expected 
liq  ut  s 

•  Testing  other  cases  not  exercised  by  the  software  developer 

Test  coverage  analyzers  and  test  data  generators  may  be  used  by  the  V&V 
agent  if  available.  Test  problems  are  reported  directly  to  the  software 
developer,  with  resolution  by  FCDSSA  if  agreement  cannot  be  reached  between 
the  V&V  agent  and  the  software  developer. 

At  the  conclusion  of  all  SP/MTs,  the  software  developer  submits  a  written 
test  report  to  FCDSSA.  This  report  shall  be  reviewed  by  the  V&V  agent  to 
determine  if  the'  SP/MT  tost  results  are  satisfactory.  He  advises  FCDSSA 
whether  the  software  developer  should  be  authorized  to  begin  FTs. 

3 . 3 . 4 . 2  Conduct  o f  FT 

The  software  developer  is  responsible  for  scheduling,  conducting ,  i 
reporting  all  FTs.  The  V&V  agent  has  the  responsibility  to  •?  *  r . » - 

conduct  of  FTs  and  independently  assess  test  status  for  FCDSSA.  ?:• 
i  software  developer  shall  supply  the  V&V  agent  with  the  latest  vers:  "t.  :  *  :.# 

internally  approved  computer  program  object  code,  in  a  form  wu.tatie  :  g 
direct  loading.  The  V&V  agent  shall  use  this  program  to  c'or.du .  *  i  :.  ie:  on  ien* 

■  evaluations  of  the  program  if  he  nudges  this  necessary  for  pr.jj  evalua¬ 
tion  of  the*  code.  Problems  found  by  the  V&V  agent  during  rest  me:  ring 

and  independent  evaluations  shall  be  reported  to  FCDSSA  on  the  Harm  form 
the  software  developer  is  usjnq  internally.  Those  reports  shall  bo  treated 
by  the;  software  developer  like  problem  reports  writte..  by  h,s  own  test 
j  *.-r  s nni.C'l . 

,V  the  conclusion  FT,  the?  software  developer  submits  i  :  rmal  *ost 
ret  or  t  i-  'ument.  ing  the  results  of  ail  FTs.  The  V&V  agent  shall  review  the 
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3 . 3 . 4 . 3  Development  of  PAT  Documentation 


PAT  involves  the  integration  of  all  PAG  functions  into  a  complete  pro¬ 
gram  that  is  tested  as  a  complete  system.  Since  PAT  requires  the  proper 
operation  of  all  functions,  it  is  not  started  until  FT  requirements  are 
completely  satisfied.  PAT  is  performed  to  accomplish  the  following: 

•  Ensure  that  the  total  man-machine  interface  is  completely  validated 

•  Ensure  proper  system  initiation,  data  entries  via  peripheral  devices, 
program  loading,  restarting,  and  the  monitoring  and  controlling  of 
PAG  operation  from  appropriate  control  stations 

•  Ensure  the  proper  interfacing  of  all  equipment  specified  in  the  PPS 

•  Ensure  the  capability  of  the  PAG  to  satisfy  all  PAG  requirements 
contained  in  the  SOD  and  PPS 

•  Ensure  the  capability  of  the  system  to  handle  erroneous  inputs 
properly  and  survive  them 

•  Ensure  the  proper  interfacing  of  all  computer  programs  specified  in 
the  IDSs 

Table  3-3  lists  the  responsibilities  for  developing  the  PAT  test  plan, 
test  specification,  test  procedures,  and  test  tools. 


Tablv  3-3.  RESPONSIBILITIES 

r  R  PAT  TEST 

DOCUMENTATION  AND  TEST  TOOLS 

Res| 

onsibi li ty 

I  tern 

fco.ju  ire.  merits 
Ana  1 y  sis 

i.)*  *vo  lof  men t. 

Testing 

Reviewing 

Approval 

T  ►  *  s  t  ;  1  ir. 

Wv  .qer.t 

V  V  , » q  e r.  t 

N/A 

FCDSSA 

NAVSEA 

M2 

St..  1  .  1 

VhV  agent 

V&V  aeon t 

N  /  A 

LCDS S A 

FCDSSA 

T-  ;  i  ■  i  .f  , 

V.,Y  agent 

VLV  agent 

N/A 

FCToSA 

FCDSSA 

TV  St.  tools 

FA  .  >  :  A 

So  f  twj  re 
i*  vo  1  oj.'or 

_ 

Software 
•iove  leper 

V&V  agent 

FCDSSA* 

*'..u  1 1 n.Mt.i.  | 
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The  PAT  test  plan  shall  define  the  total  scope  of  the  testing  to  be  per¬ 
formed,  containing  precise  statements  of  the  purpose,  scope,  and  schedule 
of  the  individual  test  being  planned.  It  shall  be  prepared  by  the  V&V  agent 
in  accordance  with  the  requirements  of  MIL-STD-1679  and  DID  DI-T-2142.  In 
addition,  a  traceability  analysis  shall  be  performed  by  the  V&V  agent  to 
ensure  that  each  requirement  in  the  PPS  and  PDS  is  covered  by  a  test  in  the 
PAT  test  plan  and  that  each  test  in  the  PAT  test  plan  is  traceable  to  a  per¬ 
formance  requirement  in  the  SOD  or  PPS.  Results  of  this  analysis  are  to  be 

reported  to  FCDSSA.  The  completed  PAT  test  plan  is  to  be  submitted  to  FCDSSA, 

who  will  review  it  for  acceptability  and  submit  it  to  NAVSEA  612  for  final 

approval . 

The  PAT  test  specification  contains  test  specifications  for  each  test 
contained  in  the  PAT  test  plan.  The  V&V  agent  shall  prepare  the  PAT  test 
specification  in  accordance  with  the  requirements  of  MIL-STD-1679  and  DID 
DI-E-2143.  Specifically  the  PAT  test  specification  shall  meet  the  following 
criteria : 

•  Each  test  in  the  PAT  test  plan  has  a  corresponding  test 
specification . 

•  Test  criteria  are  iden+  'tied. 

•  Test  methods  are  explained. 

•  The  purpose  of  each  test  is  identified. 

•  The  software  to  be  tested  and  the  scope  of  each  test  is  identified. 

•  Support  requirements ,  inputs,  required  accuracies,  expected  output, 
and  data  collection  methods  for  each  test  are  identified. 

•  System  interfaces,  method  of  data  exchange,  timing  requirements, 
degraded  operations,  casualty  recovery,  and  display  requirements  are 
ident i f i ed. 

The  PAT  test  specifications  are  submitted  to  FCDSSA  for  review  and  approval. 

The  PAT  test  procedures  give  detailed  instructions  for  test  execution 
and  for  evaluation  of  the  results  of  each  test  specified.  They  shall  be 
developed  from  the  PAT  test  specification  and  relevant  design  documents  by 
the  V&V  agent  in  accordance  with  Mil, -STD-1679  and  DID  DI-T-2144.  The  PAT 
test  procedures  oh  ill  moot  'll-  following  criteria: 

•  The  organization  or  structure  of  the  procedure  and  any  assumptions 
or  any  constraints:  r.  its  usage  are  identified. 

•  Dotai  ie-.l  instil  '  i  ■  ■  i . ;-u  the  so  tig  and  operation  of  each  test  are 
ins  out  1. 

•  Th-  tot  il  •  ■  :  ,  .  o  :  .  >v  i  ,  digital  processor  programs,  and 

suppor‘  in-:  h  m  .  i  •gw-,  led  for  ouch  operation  arc  described. 

•  The  n  j.j  i  i  •  .  a  : .  t  ,  :  ■  :•  •  •!  ■  ■  rut  i.oi .  ate  specified  if 

they  difiet  *i  c-  t ;  i  i  r .  -m<  ■  n  t  • ; . 
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•  Equipment  required  for  operation  is  identified  by  official 
nomenclature . 

•  Materials  and  personnel  required  for  the  performance  of  the  test 
are  identified. 

•  The  test  setup  and  energizing  procedures  are  given. 

•  The  program  loading  procedure  is  given. 

Completed  PAT  test  procedures  are  reviewed  and  approved  by  FCDSSA. 

Requirements  for  test  tools  used  during  PAT  are  specified  by  FCDSSA 
and  developed  by  the  software  developer,  as  shown  in  Table  3-3.  All  test 
tools  used  for  PAT  are  to  be  reviewed  and  verified  by  the  V&V  agent  and 
certified  by  FCDSSA  prior  to  completion  of  the  program-code-and-debug  phase. 
Once  the  PAT  test  tools  have  been  certified,  they  shall  be  configuration- 
managed  by  FCDSSA  in  a  manner  similar  to  that  for  the  PAG  documentation  and 
computer  programs. 

3.3.5  Status  Reports 


During  the  program-code-and-debug  phase,  the  V&V  agent  shall  submit 
monthly  status  reports  to  FCDSSA  and  NAVSEA  in  addition  to  those  submitted 
as  a  result  of  specific  activity,  such  as  document  review  or  PAT  test  docu¬ 
mentation.  These  reports  shall  contain  at  least  the  following  information: 

•  An  assessment  of  program  progress  with  respect  to  planned  schedules 

•  A  description  of  outstanding  technical  problems  requiring  resolution 

•  Reports  of  FT  test  status,  including  number  of  tests  planned, 
number  of  tests  conducted,  and  number  of  problems  reported 

•  A  summary  of  the  tasks  performed  during  the  past  month 

•  A  summary  of  tasks  planned  for  the  coming  month 


3 . 4  PROGRAM-ACCEPTANCE-TEST  PHAoE 

The  program-acceptance-test  phase  begins  with  the  establishment  of  the 
preliminary  product  baseline  and  ends  with  the  establishment  of  the  final 
product  baseline.  Figure  2-2  shows  the  activity  that  occurs  during  this 
phase.  The  object  of  this  phase  is  to  test  the  developed  product  to  ensure 
that  it  meets  all  PAG  requirements  contained  in  the  SOD  and  PPS.  The  formal 
acceptance  of  the  product  by  the  Navy  is  based  on  the  results  of  the  testing 
conducted  during  this  phase.  This  is  the  last  phase  where  the  PAG  is  tested 
as  an  entity;  the  next  phase  (the  system-test  phase)  tests  the  entire  JTIDS 
system  end-to-end,  and  the  PAG  is  only  one  component  of  the  system  under 
test . 


The  only  other  activity  that  occurs  durinq  this  phase  is  the  develop¬ 
ment  of  test  documentation  for  the  system  integration  test  (SIT)  and  Navy 
interoperability  test  (NIT),  and  the  specification,  development,  and 
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certification  of  any  test  tools  required  for  SIT  and  NIT  that  have  not 
previously  been  required. 

3.4.1  Documentation  Review 


Except  for  the  test  documentation  addressed  in  Section  3.4.4,  no  formal 
documentation  is  developed  during  this  phase.  All  PAG  specification  and 
design  documents  have  already  been  approved  and  are  under  CM.  Updates  to 
the  configuration-managed  documents  will  only  reflect  the  incorporation  of 
approved  ECPs  and  problem  resolutions.  Updated  documentation  that  is  re¬ 
vised  to  include  approved  SCNs  shall  be  reviewed  by  the  V&V  agent  to  ensure 
that  all  SCNs  are  properly  incorporated  and  are  highlighted  by  a  vertical 
black  bar  in  the  margin. 

3.4.2  Design  Reviews  and  Audits 


The  final  product  baseline  configuration  audit  (FPBCA)  is  held  at  the 
completion  of  the  program-acceptance-test  phase.  The  purpose  of  the  FPBCA 
is  to  verify  formally  that  the  PAG  computer  program  meets  the  performance 
requirements  in  the  SOD  and  the  PPS,  that  it  is  consistent  with  program 
documentation,  and  that  all  modifications  found  necessary  as  a  result  of 
PAT  have  been  properly  incorporated  into  the  program  package  and  supporting 
documentation.  Upon  satisfactory  completion  of  the  audit,  the  final  product 
baseline  is  established. 

FCDSSA  has  the  responsibility  to  schedule  and  conduct  the  FPBCA.  It 
shall  notify  attendees  by  official  correspondence  at  least  30  days  in  advance 
of  the  scheduled  date.  Attendees  at  the  FPBCA  shall  include  the  following: 

.  NA7ELEX,  PME-109 

.  NAVSEA  612 

.  FCDSSA,  San  Di •.•go 

.  Vf»V  nuor.t- 

•  So  f  tv.uri  d.  V'  !.■  .■:  "  r 

•  OFTEVFOR 


NOSC 


PA1,  IDS  interfacing  agencies 


The  latest  version  of  the  following  d< •curio!,t:-  shall  be  made  available  to  the 
FPBCA  attendees: 


•  rrs 

•  IDS 

•  DbDD 


•  PDD 
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•  Source  code  listings 

•  PAT  test  reports 

Attendees  are  required  to  review  the  PAG  computer  program  at  the  low¬ 
est  level.  Attendees  should  therefore  be  limited  to  those  individuals  with 
the  expertise  to  conduct  a  thorough  review  at  this  level.  FCDSSA  shall 
ensure  that  the  software  developer  has  key  programming  and  design  personnel 
in  attendance  or  on  call  to  provide  assistance  as  necessary.  The  V&V  agent 
has  the  responsibility  to  be  the  technical  lead  for  the  FPBCA  and  describe 
all  findings  of  the  FPBCA  audit  team  for  transmittal  by  FCDSSA  to  the  soft¬ 
ware  developer.  The  V&V  agent  shall  ensure  that  the  following  documents 
are  reviewed  in  accordance  with  the  following  criteria: 

•  PAT  test  reports  document  the  results  of  PAT  and  provide  the  basis 
for  contractual  acceptance  of  the  product  by  the  Government.  The 
audit  team  shall  ensure  the  following: 

• •  PAT  was  conducted  in  accordance  with  the  approved  PAT  test 
procedures. 

••  All  deviations  from  the  PAT  procedures  are  adequately  docu¬ 
mented,  with  supporting  rationale  to  explain  their  necessity. 

*•  All  discrepancies  and  anomalies  that  occurred  during  testing 
are  resolved  to  the  satisfaction  of  the  audit  team. 

••  The  PAT  test  report  includes  an  overall  analysis  of  the  perform¬ 
ance  of  the  PAG  with  respect  to  the  PAG  requirements  contained 
in  the  SOD  and  PPS. 

••  The  number  of  software  errors  is  within  the  limits  specified 
by  MIL-STD-1679. 

• •  The  number  of  object  patches  is  within  the  limits  specified  by 
MIL-STD-1679. 

••  Test  data  are  maintained  in  a  test  data  file  for  future  reference. 

•  The  FDD  is  reviewed  to  ensure  that  all  changes  to  the  computer  pro¬ 
gram  as  a  result  of  PAT  are  properly  contained  in  the  PDD.  The 
updated  PDD  shall  be  reviewed  to  ensure  that  it  still  meets  the 
requirements  applicable  to  the  PPBCA  contained  in  Section  3.3.2. 

•  The  program  package  document  shall  be  reviewed  to  ensure  that  all 
changes  to  the  computer  program  as  a  result  of  PAT  are  properly 
contained  in  the  program  package  document.  Criteria  contained  in 
Section  3.3.2  for  review  of  the  program  package  document  shall  be 
used  during  the  FPBCA. 

•  SIT  test  documentation:  The  SIT  validates  the  integration  of  hard¬ 
ware  and  software  of  all  JTIDS  platform  components  (terminal,  PAG, 
and  CDS)  into  a  total  system.  These  documents  are  reviewed  to 
ensure  that  the  functions  of  the  PAG  are  extensively  and  correctly 
tested  during  the  JTIDS  SIT. 
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•  NIT  test  documentations :  The  NIT  demonstrates  the  simultaneous 
interoperability  in  a  realistic  operational  environment  of  three 
JTIDS  equipped  platforms:  a  carrier  (CV) ,  an  E-2C  aircraft,  and 

an  F-14A  aircraft.  These  documents  are  reviewed  to  ensure  that  the 
functions  of  the  PAG  are  extensively  and  correctly  tested  during 
the  JTIDS  NIT. 

Minutes  of  the  FFBCA  containing  all  findings  of  the  audit  team  shall  be 
submitted  to  FCDSSA  within  two  weeks  by  the  V&V  agent.  The  V&V  agent  shall 
monitor  the  completion  of  all  post-audit  action  items  necessary  to  complete 
the  audit  satisfactorily.  When  this  is  completed,  FCDSSA  will  request 
NAVSEA  612  to  establish  the  final  product  baseline. 

3.4.3  Configuration  Management 

During  the  program- acceptance-test  phase,  all  PAG  products  are  under 
formal  CM,  including  the  program  specifications  and  design  documents  and 
the  computer  program  itself.  Procedures  and  responsibilities  for  processing 
ECPs  to  all  configuration-managed  documents  are  the  same  as  those  described 
in  Section  3.1.3.  This  is  the  first  phase,  however,  where  the  computer 
programs  have  been  under  the  formal  discipline  of  CM. 

During  PAT,  it  is  imperative  that  the  configuration  of  the  PAG  computer 
program  undergoing  test  be  known  at  any  time.  This  is  accomplished  by  plac¬ 
ing  the  computer  program  that  forms  the  basis  of  the  preliminary  product 
baseline  under  the  control  of  a  software  library  maintained  by  the  FCDSSA 
Configuration  Management  Office  (CMO) .  All  formal  tests  conducted  during 
the  program- acceptance-test  phase  are  to  be  run  with  a  computer  program 
obtained  by  the  software  librarian.  The  software  librarian  is  not  author¬ 
ized  to  change  the  computer  program  stored  in  his  library  and  has  the  respon¬ 
sibility  to  safeguard  it  against  changes,  inadvertent  or  intentional,  by 
others.  He  will  therefore  maintain  duplicate  master  records  and  make  per¬ 
iodic  comparisons  of  the  programs  checked  out  of  the  library  with  the  master 
records.  The  only  method  ot  updating  the  software  library  is  to  accept 
from  FCDSSA  an  approved,  recompiled  computer  program  or  approved  object 
patches  resulting  from  either  a  resolution  of  a  software  trouble  report  (TR) 
or  new  requirements  contained  in  an  approved  ECP. 

Software  TRs  are  written  to  record  any  discrepancy  found  during  any 
part  of  PAT,  whether  informal  testing,  dry  runs,  or  formal  testing.  TRs  may 
be  initiated  as  a  result  of  any  of  the  following: 

•  Discrepancies  in  documentation 

•  Incomplete  or  inaccurate  test  documentation 

•  Computer  program  errors 
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Software  TRs  shall  show  all  essential  data  on  each  software  problem  detected 
during  test,  including  the  following  as  applicable: 

•  Date 

•  Category  in  accordance  with  MIL-STD-1679 

•  Priority  in  accordance  with  MIL-STD-1679 

•  Control  number 

•  Title 

•  Program  designation 

•  Program  document  against  which  the  TR  is  written 

•  Site 

•  Identification  of  computer  program  under  test  (not  applicable 
for  documentation) 

•  Reference  document 

•  Function  affected 

•  Module  affected 

•  Test  step 

•  Originator 

•  Activity  and/or  code  of  originator 

•  Telephone  number  of  originator 

•  Elapsed  time  into  test  from  program  start 

•  Simulation  used 

•  Other  sites  and  programs  linked  with 

•  Transients  loaded  in  memory  when  problem  occurred 

•  Ability  of  problem  to  be  duplicated 

•  Memory  dump  data  reference 

•  Trouble  description 

•  Computer  hardware  and  register  status 

The  test  conductor  shall  have  the  authority  to  designate  the  responsible 
personnel  to  prepare  and  submit  TRs  during  any  test. 

All  TRs  shall  be  forwarded  to  the  FCDSSA  CMO  for  action  and  record¬ 
keeping  purposes.  Upon  receipt  of  a  TR,  the  CMO  will  assign  a  serial  number 
and  log  it  into  a  TR  data  base.  Duplicate  TRs  will  be  eliminated  by  the 
CMO.  The  TR  is  then  forwarded  to  the  FCDSSA  program  manager,  who  verifies 
the  priority  designated  by  the  originator,  assigns  the  responsibility  for 
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recommending  corrective  action,  and  determines  the  response  data  based  on 
the  priority.  The  following  priority  categories  shall  be  used: 

•  Priority  1  -  An  error  that  prevents  the  accomplishment  of  an  opera¬ 
tional  or  mission-essential  function  in  accordance  with  official 
requirements  (e.g.,  causes  a  program  stop),  that  affects  an  opera¬ 
tor  to  the  extent  that  the  operator  prevents  the  accomplishment  of 
an  operational  or  mission-essential  function,  or  that  jeopardizes 
personnel  safety 

•  Priority  2  -  An  error  that  adversely  affects  the  accomplishment  of 
an  operational  or  mission-essential  function  in  accordance  with 
official  requirements  so  as  to  degrade  performance  and  for  which  no 
work-around  solution  exists,  or  that  interferes  with  an  operator  to 
the  extent  that  the  operator  adversely  affects  the  accomplishment 
of  an  operational  or  mission-essential  function  so  as  to  degrade 
performance  and  for  which  no  work-around  alternative  exists  (Reload¬ 
ing  or  restarting  the  program  is  not  an  acceptable  work-around 
alternative . ) 

’  Priority  3  -  An  error  that  adversely  affects  the  accomplishment  of 
an  operational  or  mission-essential  function  in  accordance  with 
official  requirements  so  as  to  degrade  performance  and  for  which 
there  is  a  reasonable  work-around  alternative,  or  that  interferes 
with  an  operator  to  the  extent  that  the  operator  adversely  affects 
the  accomplishment  of  an  operational  or  mission-essential  function 
so  as  to  degrade  performance  and  for  which  there  is  a  reasonable 
work-around  alternative  (Reloading  or  restarting  the  program  is 
not  an  acceptable  work-around  alternative.) 

•  Priority  4  -  An  error  that  is  an  inconvenience  or  annoyance  to  the 
operator  but  does  not  affect  a  required  operational  or  mission- 
essential  function 

•  Priority  5  -  AJ 1  otner  errors 

The  FCDSSA  CMO  shall  maintain  a  historical  TR  data  base  containing  the 
following  information  about  each  TR  written  against  the  PAG  computer 
program: 

•  Identification  number 

•  Title 

•  Summary  description  problem 

•  Priority 

•  Required  resolution  date 

•  Action  assignee 

•  Actual  resolution  date 


•  Summary  of  resolution 

•  Status  (open,  closed,  withdrawn) 

The  CMO  shall  also  publish  a  weekly  written  report  on  the  status  of  all  open 
(unresolved)  TRs . 

Corrective-action  assignees  shall  recommend  one  of  the  following  types 
of  corrective  action,  or  a  corrective  action  not  listed  below  if  a  special 
situation  exists: 

•  Correction  by  object  patch 

•  Correction  by  source  recompile 

•  Correction  by  documentation  change 

•  Correction  by  ECP 

•  Waiver 

•  Withdrawal  because  problem  cannot  be  duplicated 

•  Withdrawal  because  program  was  operating  correctly 

The  V&V  agent  shall  review  all  recommended  problem  corrections  and 
advise  FCDSSA  on  their  acceptability.  He  shall  monitor  the  extent  of  object 
patches  in  the  system  with  respect  to  the  limits  established  by  MIL-STD-1679 
and  recommend  when  all  or  part  of  the  computer  program  must  be  recompiled  to 
eliminate  object  patches.  He  shall  also  identify  the  specific  corrections 
recommended  for  inclusion  in  the  recompile. 

When  FCDSSA  approves  a  problem  correction  that  requires  either 
the  development  of  an  object  patch  or  a  recompile  of  the  computer  program 
by  the  software  developer,  the  V&V  agent  shall  determine  the  amount  of  retest 
or  regression  testing  required  before  delivery  of  the  patch  or  recompiled 
program  to  FCDSSA,  witness  the  required  testing,  and  evaluate  the  test  re¬ 
sults.  Upon  satisfactory  completion  of  the  required  tasting,  the  FCDSSA 
program  manager  shall  authorize  the  software  library  to  accept  the  recompiled 
computer  program  or  object  patches. 

3.4.4  Test  and  Evaluation 

As  shown  in  Figure  2-2,  the  test  and  evaluation  activity  during  the 
program  acceptance  test  phase  is  as  follows: 

•  Conduct  of  PAT 

•  Development  of  SIT  and  NIT  test  documentation 

•  Specification,  development,  and  certification  of  test  tools  for 
SIT  and  NIT 

The  purpose  of  the  PAT  is  to  ensure  that  the  PAG  satisfies  all  the 
requirements  of  the  SOD  and  PPS.  The  interaction  of  all  component  functions 


shall  be  tested.  The  operating  capabilities  to  be  demonstrated  include  the 
following : 

•  Program  loading  and  initialization 

•  Interface  communication  and  message  traffic  exchange 

•  System  capacities,  accuracies,  and  limitations 

•  Visual  display  indicators 

•  Data  entry 

•  Operator  interface 

•  Data  processing  and  output 

•  Degraded  modes  and  system  recovery 

FCDSSA  is  responsible  for  scheduling  and  directing  all  the  test  activi¬ 
ties  associated  with  PAT.  The  V&V  agent  is  to  act  as  test  conductor  and 
report  the  results  of  PAT.  Formal  PAT  is  to  be  conducted  in  accordance  with 
approved  PAT  test  plans,  test  specifications,  and  test  procedures  using  con¬ 
figuration-managed  software  obtained  from  the  software  library.  Formal  PAT 
may  be  preceded,  however,  by  informal  testing  using  unapproved  test  pro¬ 
cedures  and  unapproved  object  patches  to  facilitate  the  resolution  of 
problems . 

At  the  conclusion  of  PAT,  the  V&V  agent  shall  submit  a  formal  test  report 
to  FCDSSA  documenting  the  results  of  PAT  and  containing  the  following: 

•  A  report  of  the  conduct  of  each  test  included  in  the  PAT  test 
specification 

•  A  report  on  whether  PAT  acceptance  criteria  have  been  met 

•  A  list  of  open  Tks  requiring  resolution  prior  to  SIT 

•  A  report  on  whether  a  twenty-percent  reserve  exists  for  total 
system  memory,  input  and  output  channels,  and  processing  time 

•  A  report  on  whether  the  number  of  software  errors  is  within  the 
limits  specified  by  Mil. -STD-1679. 

•  A  report  on  whether  the  number  of  object  patches  is  within  the 
limit  specified  in  MIL-STD-1679. 

The  purpose  of  SIT  is  twofold.  First,  SIT  validates  the  integration 
of  hardware  and  software  of  all  JTIDS  platform  component s  (terminal,  PAG, 
and  CDS)  into  a  complete  platform  system  consisting  of  the  hardware  and 
software  of  the  JTIDS  terminal,  the  PAG,  and  the  CDS.  SIT  includes  such 
tests  as  validation  of  the  total  platform  man-machine  interface  and  vali¬ 
dation  of  system  initiation,  program  loading,  and  controlling  of  system 
operation.  Second,  SIT  validates  the  system  capability  of  JTIDS  against 
the  requirements  of  the  SOS  and  SOD.  This  includes  validation  of  the  capa¬ 
bility  to  perform  the  following  functions: 

•  Transmit  information  received  rrom  the  host  platform  CDS  in  the 
prone r  format 


J-  is 


Forward  properly  formatted  messages  received  from  an  external 
source  to  the  host  platform  CDS 


Table  3-4  lists  the  responsibilities  for  developing  the  SIT  test  plan,  test 
specification,  test  procedures,  and  test  tools. 


Table  3-4.  RESPONSIBILITIES  FOR  SIT  TEST  DOCUMENTATION  AND  TEST  TOOLS 

Responsibility 

Item 

Requirements 

Analysis 

Development 

Testing 

Reviewing 

Approval 

Test  plan 

FCDSSA* 

FCDSSA* 

N/A 

NAVSEA  612 

PME-109 

Test 

specification 

FCDSSA* 

FCDSSA* 

N/A 

NAVSEA  612 

PME-109 

Test  procedures 

FCDSSA* 

FCDSSA* 

N/A 

NAVSEA  612 

PME-109 

Test  tools 

FCDSSA* 

FCDSSA* 

FCDSSA* 

NAVSEA  612 

PME-109 

•Assisted  by  V&V 

agent . 

The  SIT  test  plan  will  define  the  total  scope  of  the  testing  to  be  per¬ 
formed,  containing  precise  statements  of  the  purpose,  scope,  and  schedule 
for  the  individual  tests  being  planned.  Specific  testing  shall  include  the 
following: 

•  Connectivity 

•  Message  selection  and  addressing 

•  Relative  navigation 

•  Subscriber  functions 

•  Secure  voice 

•  Platform  CDS  to  PAG  to  terminal  translation,  protocol,  and 
buffering 

The  V&V  agent  shall  assist  FCDSSA  in  ensuring  that  system  tests  are  defined 
that  test  the  PAG  computer  programs  to  the  maximum  practical  extent. 

The  SIT  test  specification  contains  test  specifications  for  each  test 
contained  in  the  SIT  test  plan.  The  Vs, V  agent  shall  assist  FCDSSA  by  review¬ 
ing  test  specifications  that  involve  the  PAG  and  by  preparing  technical 
inputs  to  the  SIT  test  specifications  as  required. 
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The  SIT  test:  procedures  contain  detailed  instructions  for  test  execu¬ 
tion  and  evaluation  of  the  results  of  each  test  specified.  The  V&V  agent 
shell  assist  FCDSSA  by  preparing  detailed  test  procedures  for  those  tests 
involving  PAG  operation  and  by  reviewing  ail  test  procedures  to  ensure  that 
PAG  operations  are  properly  identified. 

The  purpose  of  NIT  is  to  demonstrate,  in  a  realistic  operational  envi¬ 
ronment,  the  simultaneous  interoperability  of  F-14A,  E-2C,  and  CV  platforms 
equipped  with  JTIDS.  NIT  will  validate  the  capability  of  the  JTIDS  plat¬ 
forms  to  interoperate  and  perform  the  following  functions: 

•  Transfer  infoim.it  > cn 

•  Compute  and  transmit  own  relative  position 

•  Identify  targets 

■  i roc ass  and  disseminate  tactical  data 


Table  1-5  lists  the  i esponsibil it  res  for  developing  the  NIT  test 
specification,  test  procedures,  and  test  tools.  The  role  of  the 
during  the  development  of  the  NIT  tost  documentation  is  the  same 
the  development  of  the  SIT  test  documentation:  assisting  FCDSSA 
ing  technical  input  and  review  to  ensure  that  FAG  is  extensively 
correctly  tested. 


plan,  test 
V&V  agent 
as  during 
by  provid- 
and 


Requirements  for  test  tools  us:  1  dui  in SIT  are  specified  and  developed 
by  FCDSSA.  All  test  fools  involving  the  i  A  ■  chill  be  verified  by  the  V&V 
agent  prior  to  certification  by  i CDSSA.  Once  the  SIT  and  NIT  test  tools 
have  been  certified,  they  snail  b<  con  f igui at  ion-managed  by  FCDSSA  like  the 
FAG  documentation  and  computer  pi  on  rams. 
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3.1.5  Status  Reports 


During  the  program-acceptance-test  phase,  the  V&V  agent  shall  submit 
monthly  status  reports  to  FCDSSA  and  NAVSEA,  in  addition  to  those  submitted 
as  a  result  of  specific  activities,  such  as  document  review  or  PAT  test 
plan  development.  They  will  contain  at  least  the  following  information: 

•  An  assessment  of  program  progress  with  respect  to  planned  schedules 

•  A  description  of  outstanding  technical  problems  requiring  resolution 

•  Reports  of  PAT  test  status,  including  number  of  tests  planned, 
number  of  tests  conducted,  and  number  of  TRs  reported 

•  A  summary  of  the  tasks  performed  during  the  past  month 

•  A  summary  of  tasks  planned  for  the  coming  month 

3.5  SYSTEM-TEST  PHASE 

The  system-test  phase  begins  with  the  establishment  of  the  final  product 
baseline  and  ends  with  the  establishment  of  the  operational  baseline.  At 
this  point  the  JTIDS  system  is  ready  to  be  deployed.  The  object  of  this 
phase  is  to  test  the  integration  of  the  hardware  and  software  of  all  the 
JTIDS  components  and  validate  the  interoperability  of  JTIDS-equipped  plat¬ 
forms.  This  testing  validates  that  the  complete  JTIDS  system  can  meet  the 
requirements  contained  in  the  TOR  and  IDR.  During  this  phase  of  testing  the 
PAG  is  not  tested  as  an  entity,  but  only  as  a  unit  of  the  overall  system 
being  tested.  Consequently,  personnel  associated  with  PAG  development  and 
test  have  only  a  support  role  in  this  pause. 

The  activity  that  occurs  during  the  system-test  phase  is  shown  m 
Figure  2-2  is  follows: 

•  Conduct  of  SIT 

•  Conduct  of  NIT 

•  Technical  evaluation  (TECHEVAL) 

•  Operational  evaluation  (OPEVAL) 

3.5.1  Documents t ion  Review 

.'Jo  forma!  documentation  in  dev.'lo:  «.-d  during  the  system-test  phase. 

All  PA1",  ,  ci  f  lent  ions  and  design  documents  have  already  boon  approved 

and  a*-.  •  und-  r  I'pdatos  to  the  conf  i  duration-managed  documents  will 

only  r-  •  f  1  •  •  t  th-  incur;  oration  of  apj  roved  ECTs  and  TR  resolutions.  Up¬ 
dat’d  e-umi  nfat  ion  that  is  revised  to  include  approved  SCNs  and  TRs  shall 
O'  i'  vi-  d  i  y  t : : •  ■  V&V  agent  to  •  nsur-  that  all  SCNs;  are  properly  incor- 
:  ora?  •  d  and  at'-  hi  mi  minted  by  a  vertical  black  bar  in  the  margin. 
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The  operational  baseline  con fiquratior.  audit 
completion  of  the  system- tost  phase.  The  purpose 
the  results  of  SIT,  NIT,  TKClit'VAP,  and  Of  KVAI.  and 
svstem  meets  the  reauireraent.s  of  the  TOR  and  TDK. 
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the  OBCA,  the  operational  baseline  will  be  established  and  the  sys- 
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NAVE!.  EX,  PME-i'O  has  the  responsibility  to  schedule  and  conduct  the 
OBCA.  In  co:  In -tine  the  OnCA,  it  will  follow  the  sumo  general  policies  and 
l  rocedurcs  U  fined,  for  KCDSSA  during  the  FPCBA,  as  contained  in  Section 
3.4..'.  Th*'-  V  .  V  aqi  nt  .cal  assist  in  the  review  if  tin-  following  documents 

is  m i  ••  *:•;>! y  to  *  he  s' Ao  computer  programs: 
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.  a  i.  ii  •  . :  <  :  . :.  •*  .  ■■  '.4.  '.  Ti.i  r-.  cere,  changes  to  the 

: :  c  ’  ;i"  •  ■  :v.  c.  c  •  lice:  if'!  nr  TK  resolution. 


;  ‘o  i  •  vf-w  1  >'■  nmnendod  corrective 

.  ivi.ji.  i'-.T-fA  >-  their  -accept  abi]  l  ty .  The  extent  of 

:  :i  Me  ;  A  •  :nmj  ;if  .-i  :  rmtr.arn  shall  also  be  monitored 

•:  .  :  ■ ' ;  .  )-•  oiiirn  i..i  c  inns  to  recompile  as  the 

. 'Ti  if.  k  crouch'  d.  When  a  recompile  recom- 

i-c  -  ‘  iiial.l  pi  '  ,  •  ••  t  mouat  of  regression  test- 
id  i  1  :■  r« 'Compile  Shull  be  up;  icvi-d  by  NAVSEA  612 

.  i  .•  ,  ,..  :  tin.  V.-.V  u  B  at  cl, all  cin-lu-u  the  necessary 

1  ■  .  i ,  ),y  fli<  t  f  W.irt  leveloper.  Test,  results 
.  ,  V.:  .  ::  , 1  i  ■  to.  up:  :  n\M  i  ■  d  NAVSEA  612 

■  •  u-  {lii-i ry  ;■  im-;  t  c  -.i.v,  •  d  -omputer  program 


;•  ;-  i  nt  •  ,  •  a,  •.  sr  and  evalua?  ion  activity  during  the 


TECHKVAL 


•  OPEVAL 


The  purpose  of  SIT  is  to  validate  the  integration  of  hardware  and  soft¬ 
ware  of  all  .ITIDS  platform  components  and  n  validate  the  system  capability 
of  JTIDS  aqainst  the  requirements  contained  in  the  SOS  and  SOD.  SIT  will  be 
conducted  in  the  integrated  combat  system  test  facility  (ICSTF).  NAVELEX , 
PME-109  is  responsible  for  the  conduct  ot  SIT.  The  V f,V  agent  shall  witness 
all  testing  to  assess  the  ; or to finance  o!  the  'AO  during  SIT  and  assist  in 
conducting  tests  involving  the  IAC. 

The  purpose  of  NTT  i :  t  :■  tv.  rat'-,  , .1  realistic  operational  envi¬ 
ronment,  the  simultaneous  :  1  : 1 t  >  f  P-1-1A,  F.-2C,  and  CV  platforms 

equipped  with  .ITIliS .  NIT  w:l  !  :  ;  ted  1:.  the  D'STF .  NAVELEX,  PME-109 

is  responsible-  for  the  c.-i.:  ;  ’  7 t.V  agent  shall  witness  all  test¬ 
ing  to  assess-  the  ■  ;  •  .  i  got  ;  NIT  issist  it.  conducting 

tests  involving  tie-  i  A  -. 


TECHEVA:  .'err,;  : 

TOR  and  lap-  at.  1  it  t  -  -  i  . t 
of  shore- :>ase«i  :  aier.it  : 
are  tested  for  1  ivt  .>•  ■ 

t  ions.  TECHEVAL  <■:.  t  :  1 
and  the  ease  with  whi 
ally  support  tin-  JTr:  :  ; 

equipment . 


the  requirements  of  the 
no icted  in  a  combination 
The  JTIDS  components 
-  .  i-  -operational  condi- 
:na 1  and  technical  design 
,  maintain,  and  logistic- 
■  the  platform  suite  of 


NAVELEX,  ;  MI.-  I'D  it  : 
documentation,  cc-nauct.,  a:,  i 
(COMOPTEVE-  1-'  1  f  spots  it  1 


-lit,  ; s  eparation  of  test 
Commander ,  7PTEVF0R 
N  ivy  platform  or  shore- 

"•  :  ■  f  -  •  : av-i  {  ' :  -.1  -. I- 

.  u .  lot.  o  ‘  .  1.  ■  It  '.Mi., 

rni.-u:  '•  •  of  !  Ail ,  and  ;  1  ■  -- 
A  i  I.  .1  w  ill  -  v.a :  ua  f .  -  i  t  :■ 
,r  1  t  .1  r.  ;  ort  t  >  EME-l  Id. 


L  r  1 1 , 


aoi  Ilf' 


ai  lilt  y  o! 

in  t. ho  TO! 


s  to  evaluate  x  h*  mission  c*f  fi.-et  1  veness,  suit- 
t  he  JTIDS  system  to  meet  the  per  forman.ee  require- 
IE-R. 


'Hi.VAL  will  begin  when  I  ML- 1  •-  ".>  judqr-s  the  syc  torn  ready  for  Of  SVAt.  and 
will  ;.><•  e.  iucteil  ot,  1  <  >.  rd  !  he  best  platforms.  COMOPTKVFOR  is  responsible 
for  -  '••  rail  iIKVAi  execution  and  will  dt vo I oj >  Hit-  OPEVAL  test  documentation, 
coord i t  ste  the  a -t  1  v  1  f  1  os  of  the  .'TIDE  test  platform.-,,  coordinate  the  test 
rendu:'  ,  and,  report  th  1  results  to  ONf  a  PME-109  shall  provide  all  personnel 
n*  i  f  1  j  1. stall  arc!  .  ■  »-t  i  fy  all  equipment  s  and  computer  proqrams  and  will 
provide  f  >  chnirnl  cii-rrrt  for  the  .ITIDS  equipment  an  1  computer  proqrams . 

Wit  ness  inq  of  OPEVAL  by  the  VkV  will  f.e  as  determined  by  PME-109. 


5. 5  Status  Reports 

Durinq  the  system- test  ; ■ha.  < -,  the  Vt»V  agent  shall  submit  monthly  status 
reports  to  FCDSSA  and  NAVSKA,  i addition  tc  those  submitted  as  a  result  of 
specific  activities,  such  as  an  EC!  review.  They  will  contain  the 
fol lowinq : 

•  An  assessment  of  program  progress  with  respect  to  planned  schedules 

•  A  description  ot  outstanding  technical  problems  requiring  resolution 

■  Reports  of  SIT,  NIT,  and  TECHEVAI.  test  status  as  applicable  to  the 
PAG,  including  number  of  tests  planned,  number  of  tests  conducted, 
and  number  of  TRs  reported 

•  A  summary  of  the  tasks  performed  during  the  past  month 

•  A  summary  of  tasks  planned  for  the  coming  month 

3.6  MAINTENANCE  PHASE 

The  maintenance  phase  begins  with  the  establishment  of  the  operational 
baseline  at  the  conclusion  of  the  OBCA.  The  JTIDS  system  is  not  deployed 
operationally  as  full  production  begins. 

Although  development  has  been  completed,  there  is  still  a  need  for 
software  ,'A  activity  in  the  maintenance  phase.  Previously  undiscovered 
problems  wail  normally  bo  encountered  : >y  operational  forces.  The  need  for 
changes  to  the  system  foment  s  :na’;  become  more  apparent  when  the  system 

is  dep loved  than  while  it  is  undergoing  testing.  Thus,  error  corrections 
and  incorporation  of  now  r  -  m  ir-  -merit will  cause  the  PAG  computer  programs 
to  be  n.od.5  i ed  . 

I'ho  i.o'  .  to  maintain  the  PA  .  computer  programs  under  CM  is  not  reduced 
rter./lv  .:.m  ..  n.-.e  the  •omputer  programs  are  no  longer  under  development.  To 
tr:d  n-  •  v  :  qur  it  ; .  of  the  PAG  computer  programs  properly,  the  changes 
to  ‘in  ■  a..  it  ion  i !  :  ;a"  should  be  controlled  as  rigidly  as  changes  to 

previ  o i  use  1  men,  ’!'!  j  regain  *h,v  changes  to  the  PAG  computer  programs 

ho  m  rie  o',  p.-  m  r-  a  :  ■  a  -  .in  . . r ;  roved  TP.  (or  its  equivalent)  or  ECP.  The 

;  no-  rj  r  a  .  r-  ve  i  op-w.-ct  pat  "lies  into  the  PAG  computer  programs  by 

o;.  r  c  .  ■  .!  •  ••  :  !  be  rr.<  d  uy  future  Navy  policy  decisions,  but 

tii'/i !  ■:  :  ■  >■  i  m-  ;  !■  n  v  si  tua  lions. 

:  c ,  •  i  :ko  t'n  .t  should  he  performed  durinq  this  phase  will 


a  'll'- 


i  ■  ‘  :  1  • -v.ilu.it  i  n  •  :  proposed  changes 

■  c,  r;  t.-.-e  of  :  ■  ;  i  ,■ !  i  -  audit  c  to  eiisun  CM  integrity  of  the 

‘  tat  i  on.i  1  ;  ■  1  in- 


Since  these  QA  tasks  will  likely  be  performed  by  organic  Navy  personnel, 
specific  requirements  for  task  performance  will  not  be  specified  by  this 
plan.  In  general,  however,  they  should  be  performed  as  carefully  during 
the  operational  life  of  the  PAG  as  during  development. 


;;--r 


